[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:53 dist-test-slave-pqr8 krb5kdc[3826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:53 dist-test-slave-pqr8 krb5kdc[3826](info): set up 2 sockets
Sep 27 21:32:53 dist-test-slave-pqr8 krb5kdc[3826](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 21:32:54.454540 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:32:54 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:32:54.621606 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:39495
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:39495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:54.922858 3839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:54.923164 3839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:54.923249 3839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:54.930298 3839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:54.930390 3839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:54.930431 3839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:54.930483 3839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:54.930523 3839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:54.939397 3839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:39495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:39495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:54.940407 3839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:54.941524 3839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:54.953076 3845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:54.953096 3847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:54.953747 3839 server_base.cc:1034] running on GCE node
W20260927 21:32:54.953842 3844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:54.954671 3839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:54.955899 3839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:54.957111 3839 hybrid_clock.cc:648] HybridClock initialized: now 1790544774957027 us; error 94 us; skew 500 ppm
I20260927 21:32:54.957422 3839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:54 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:54.960563 3839 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:32:54.961831 3839 webserver.cc:460] Webserver started at http://127.3.186.254:39807/ using document root <none> and password file <none>
I20260927 21:32:54.962405 3839 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:54.962496 3839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:54.962718 3839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:54.965266 3839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "510bd03ed301452bafd91225fa872d7f"
format_stamp: "Formatted at 2026-09-27 21:32:54 on dist-test-slave-pqr8"
server_key: "d70b225881577a01a208ebf324891f33"
server_key_iv: "46229725b94c0f27fe9ea2828dcd1eb0"
server_key_version: "encryptionkey@0"
I20260927 21:32:54.965968 3839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "510bd03ed301452bafd91225fa872d7f"
format_stamp: "Formatted at 2026-09-27 21:32:54 on dist-test-slave-pqr8"
server_key: "d70b225881577a01a208ebf324891f33"
server_key_iv: "46229725b94c0f27fe9ea2828dcd1eb0"
server_key_version: "encryptionkey@0"
I20260927 21:32:54.970548 3839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:32:54.973438 3854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:54.974674 3839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:32:54.974846 3839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "510bd03ed301452bafd91225fa872d7f"
format_stamp: "Formatted at 2026-09-27 21:32:54 on dist-test-slave-pqr8"
server_key: "d70b225881577a01a208ebf324891f33"
server_key_iv: "46229725b94c0f27fe9ea2828dcd1eb0"
server_key_version: "encryptionkey@0"
I20260927 21:32:54.975028 3839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:54.992903 3839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:54.996254 3839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:54.996531 3839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:55.014417 3839 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:39495
I20260927 21:32:55.014608 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:39495 every 8 connection(s)
I20260927 21:32:55.015604 3839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:55.018853 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:55.025486 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 3839
I20260927 21:32:55.025812 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:32:55.027997 3819 external_mini_cluster.cc:1428] Setting key fd210872ab7d502b8822c1d90ea33519
I20260927 21:32:55.029256 3906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f: Bootstrap starting.
I20260927 21:32:55.033105 3906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:55.034137 3906 log.cc:824] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:55.036598 3906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f: No bootstrap required, opened a new log
I20260927 21:32:55.044742 3906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510bd03ed301452bafd91225fa872d7f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 39495 } }
I20260927 21:32:55.045138 3906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:55.045257 3906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510bd03ed301452bafd91225fa872d7f, State: Initialized, Role: FOLLOWER
I20260927 21:32:55.045807 3906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510bd03ed301452bafd91225fa872d7f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 39495 } }
I20260927 21:32:55.046032 3906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:55.046175 3906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:55.046327 3906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:55.048451 3906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510bd03ed301452bafd91225fa872d7f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 39495 } }
I20260927 21:32:55.048964 3906 leader_election.cc:302] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510bd03ed301452bafd91225fa872d7f; no voters:
I20260927 21:32:55.049583 3906 leader_election.cc:288] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:55.049698 3911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:55.051522 3911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [term 1 LEADER]: Becoming Leader. State: Replica: 510bd03ed301452bafd91225fa872d7f, State: Running, Role: LEADER
I20260927 21:32:55.052106 3906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:55.052044 3911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510bd03ed301452bafd91225fa872d7f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 39495 } }
I20260927 21:32:55.055027 3912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510bd03ed301452bafd91225fa872d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510bd03ed301452bafd91225fa872d7f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 39495 } } }
I20260927 21:32:55.055503 3912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:55.055891 3913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510bd03ed301452bafd91225fa872d7f. Latest consensus state: current_term: 1 leader_uuid: "510bd03ed301452bafd91225fa872d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510bd03ed301452bafd91225fa872d7f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 39495 } } }
I20260927 21:32:55.056239 3913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:55.058102 3918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 21:32:55 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:32:55.064007 3918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:55.074064 3918 catalog_manager.cc:1347] Generated new cluster ID: 44b4bed7e81444d9aa86bb56e45313d5
I20260927 21:32:55.074193 3918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:55.074895 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:55.032170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.3.186.254:39495)
0927 21:32:55.033424 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:55.033441 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:55.034703 (+ 1262us) server_negotiation.cc:407] Connection header received
0927 21:32:55.044173 (+ 9470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:55.044196 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:55.044605 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:55.044974 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:55.047839 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:55.049811 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:55.052987 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:55.054364 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:55.054588 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:55.063742 (+ 9154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:55.063755 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:55.063783 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:55.063853 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:55.069739 (+ 5886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:55.070748 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:55.070756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:55.070766 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:55.070944 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:55.071459 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:55.071464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:55.071467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:55.072088 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:55.072410 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:55.073176 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 21:32:55.073604 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":795,"threads_started":1}
I20260927 21:32:55.089635 3918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:55.090701 3918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:55.095317 3918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 510bd03ed301452bafd91225fa872d7f: Generated new TSK 0
I20260927 21:32:55.096163 3918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:32:55.135161 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:39495
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:55.444905 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:55.445133 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:55.445279 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:55.451978 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:55.452059 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:55.452225 3932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:32:55.461613 3932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:39495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:55.462661 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:55.463850 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:55.476156 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:55.476220 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:55.476127 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:55.477130 3932 server_base.cc:1034] running on GCE node
I20260927 21:32:55.477754 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:55.478560 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:55.479740 3932 hybrid_clock.cc:648] HybridClock initialized: now 1790544775479689 us; error 51 us; skew 500 ppm
I20260927 21:32:55.480022 3932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:55 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:55.482810 3932 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:32:55.484054 3932 webserver.cc:460] Webserver started at http://127.3.186.193:44415/ using document root <none> and password file <none>
I20260927 21:32:55.484664 3932 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:55.484799 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:55.485088 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:55.487594 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "21d09a48d60c449fb0fe2d7695593104"
format_stamp: "Formatted at 2026-09-27 21:32:55 on dist-test-slave-pqr8"
server_key: "40414e0c5882358e3944390097ca039c"
server_key_iv: "3528cc65ba22bf59541ae7397aefb760"
server_key_version: "encryptionkey@0"
I20260927 21:32:55.488262 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "21d09a48d60c449fb0fe2d7695593104"
format_stamp: "Formatted at 2026-09-27 21:32:55 on dist-test-slave-pqr8"
server_key: "40414e0c5882358e3944390097ca039c"
server_key_iv: "3528cc65ba22bf59541ae7397aefb760"
server_key_version: "encryptionkey@0"
I20260927 21:32:55.493026 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 21:32:55.495998 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:55.497098 3932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:32:55.497247 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "21d09a48d60c449fb0fe2d7695593104"
format_stamp: "Formatted at 2026-09-27 21:32:55 on dist-test-slave-pqr8"
server_key: "40414e0c5882358e3944390097ca039c"
server_key_iv: "3528cc65ba22bf59541ae7397aefb760"
server_key_version: "encryptionkey@0"
I20260927 21:32:55.497426 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:55.512998 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:55.516359 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:55.516976 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:55.517779 3932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:32:55.518896 3932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:32:55.518992 3932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:55.519145 3932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:32:55.519232 3932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:55.551568 3932 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:38529
I20260927 21:32:55.551642 4059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:38529 every 8 connection(s)
I20260927 21:32:55.552776 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:32:55.558663 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 3932
I20260927 21:32:55.558861 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:32:55.559480 3819 external_mini_cluster.cc:1428] Setting key 6a6b642672a81fa4136e132abde029b6
Sep 27 21:32:55 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:32:55.571524 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:55.554997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:58457 (local address 127.3.186.254:39495)
0927 21:32:55.555176 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:55.555180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:55.556148 (+ 968us) server_negotiation.cc:407] Connection header received
0927 21:32:55.557844 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:55.557850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:55.557932 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:55.558040 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:55.559543 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:55.560385 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:55.562423 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:55.562851 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:55.562981 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:55.566221 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:55.566226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:55.566229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:55.566257 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:55.568712 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:55.569366 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:55.569375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:55.569378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:55.569480 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:55.569910 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:55.569915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:55.569918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:55.570264 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:55.570413 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:55.570998 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 21:32:55.571204 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:32:55.572306 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:55.555398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:39495 (local address 127.3.186.193:58457)
0927 21:32:55.556010 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:55.556043 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:55.557596 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:55.558270 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:55.558278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:55.558823 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:55.559358 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:55.559396 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:55.560518 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:55.560525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:55.562256 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:55.562268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:55.563057 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:55.563064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:55.563277 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:55.563938 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:55.563961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:55.566018 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:55.568844 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:55.568852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:55.568865 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:55.569236 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:55.569618 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:55.569623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:55.569627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:55.569790 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:55.570386 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:55.570393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:55.570677 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 21:32:55.570916 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 21:32:55.571339 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":229,"threads_started":1}
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
I20260927 21:32:55.573998 4060 heartbeater.cc:347] Connected to a master server at 127.3.186.254:39495
I20260927 21:32:55.574271 4060 heartbeater.cc:464] Registering TS with master...
I20260927 21:32:55.574970 4060 heartbeater.cc:511] Master 127.3.186.254:39495 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
I20260927 21:32:55.577135 3871 ts_manager.cc:194] Registered new tserver with Master: 21d09a48d60c449fb0fe2d7695593104 (127.3.186.193:38529)
I20260927 21:32:55.578642 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:58457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:32:55.595774 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:39495
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:55.895890 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:55.896112 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:55.896272 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:55.903259 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:55.903365 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:55.903537 4065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:32:55.912235 4065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:39495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:55.913252 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:55.914410 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:55.926916 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:55.926951 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:55.926959 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:55.927533 4065 server_base.cc:1034] running on GCE node
I20260927 21:32:55.927995 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:55.928722 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:55.929915 4065 hybrid_clock.cc:648] HybridClock initialized: now 1790544775929875 us; error 33 us; skew 500 ppm
I20260927 21:32:55.930147 4065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:55 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:55.932832 4065 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:32:55.934100 4065 webserver.cc:460] Webserver started at http://127.3.186.194:39273/ using document root <none> and password file <none>
I20260927 21:32:55.934703 4065 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:55.934844 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:55.935153 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:55.937758 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "5576670202814d70a42e2b9e2ae40838"
format_stamp: "Formatted at 2026-09-27 21:32:55 on dist-test-slave-pqr8"
server_key: "d95c8f6961b8f9159ad711d7017b5aaa"
server_key_iv: "68ce61f05e9b3f361205f00b6f775872"
server_key_version: "encryptionkey@0"
I20260927 21:32:55.938454 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "5576670202814d70a42e2b9e2ae40838"
format_stamp: "Formatted at 2026-09-27 21:32:55 on dist-test-slave-pqr8"
server_key: "d95c8f6961b8f9159ad711d7017b5aaa"
server_key_iv: "68ce61f05e9b3f361205f00b6f775872"
server_key_version: "encryptionkey@0"
I20260927 21:32:55.943269 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:32:55.946300 4080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:55.947448 4065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:32:55.947602 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "5576670202814d70a42e2b9e2ae40838"
format_stamp: "Formatted at 2026-09-27 21:32:55 on dist-test-slave-pqr8"
server_key: "d95c8f6961b8f9159ad711d7017b5aaa"
server_key_iv: "68ce61f05e9b3f361205f00b6f775872"
server_key_version: "encryptionkey@0"
I20260927 21:32:55.947790 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:55.984954 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:55.988327 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:55.988581 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:55.989810 4065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:32:55.990880 4065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:32:55.990950 4065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:55.991125 4065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:32:55.991171 4065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:56.025002 4065 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:46539
I20260927 21:32:56.025036 4192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:46539 every 8 connection(s)
I20260927 21:32:56.026166 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:32:56.030354 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4065
I20260927 21:32:56.030527 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:32:56.031097 3819 external_mini_cluster.cc:1428] Setting key f376a5434b92d33fb0fd3bfd2b517080
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:32:56.049634 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.028566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:43839 (local address 127.3.186.254:39495)
0927 21:32:56.028775 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.028782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.031275 (+ 2493us) server_negotiation.cc:407] Connection header received
0927 21:32:56.033558 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.033568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.033660 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.033762 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.036098 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.036809 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.038751 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.039271 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.039421 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.043662 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.043668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.043673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.043715 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.046477 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.047162 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.047167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.047170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.047319 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.047712 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.047717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.047720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.048234 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.048380 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.049014 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.049209 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:32:56.050690 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.028990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:39495 (local address 127.3.186.194:43839)
0927 21:32:56.031130 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:56.031179 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:56.033257 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:56.034019 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:56.034031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:56.034765 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:56.035916 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:56.035943 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.036972 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.036980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:56.038614 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:56.038625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.039511 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.039518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:56.039784 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.040598 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:56.040631 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:56.043469 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:56.046631 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:56.046639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:56.046653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:56.047038 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:56.047426 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:56.047431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:56.047435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:56.047599 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:56.048357 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:56.048363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:56.048665 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 21:32:56.048907 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 21:32:56.049358 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1845,"thread_start_us":1686,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
I20260927 21:32:56.052443 4193 heartbeater.cc:347] Connected to a master server at 127.3.186.254:39495
I20260927 21:32:56.052793 4193 heartbeater.cc:464] Registering TS with master...
I20260927 21:32:56.053671 4193 heartbeater.cc:511] Master 127.3.186.254:39495 requested a full tablet report, sending...
I20260927 21:32:56.055123 3871 ts_manager.cc:194] Registered new tserver with Master: 5576670202814d70a42e2b9e2ae40838 (127.3.186.194:46539)
I20260927 21:32:56.055986 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:43839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:32:56.071993 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:39495
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:56.373689 4198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:56.373927 4198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:56.374011 4198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:56.380895 4198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:56.380988 4198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:56.381088 4198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:32:56.389729 4198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:39495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:56.390667 4198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:56.392169 4198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:56.404424 4203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:56.404423 4204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:56.404455 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:56.405136 4198 server_base.cc:1034] running on GCE node
I20260927 21:32:56.405655 4198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:56.406425 4198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:56.407608 4198 hybrid_clock.cc:648] HybridClock initialized: now 1790544776407575 us; error 31 us; skew 500 ppm
I20260927 21:32:56.407886 4198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:56.411070 4198 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:32:56.412376 4198 webserver.cc:460] Webserver started at http://127.3.186.195:44637/ using document root <none> and password file <none>
I20260927 21:32:56.412976 4198 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:56.413098 4198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:56.413385 4198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:56.415966 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "63822de4be414026a663b71cfa92b09e"
format_stamp: "Formatted at 2026-09-27 21:32:56 on dist-test-slave-pqr8"
server_key: "4aa9235e066968bde09477b00944dbe5"
server_key_iv: "8bddc5f26feae2299699d0fc1dc8b295"
server_key_version: "encryptionkey@0"
I20260927 21:32:56.416643 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "63822de4be414026a663b71cfa92b09e"
format_stamp: "Formatted at 2026-09-27 21:32:56 on dist-test-slave-pqr8"
server_key: "4aa9235e066968bde09477b00944dbe5"
server_key_iv: "8bddc5f26feae2299699d0fc1dc8b295"
server_key_version: "encryptionkey@0"
I20260927 21:32:56.421121 4198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 21:32:56.424268 4213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:56.425419 4198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:32:56.425601 4198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "63822de4be414026a663b71cfa92b09e"
format_stamp: "Formatted at 2026-09-27 21:32:56 on dist-test-slave-pqr8"
server_key: "4aa9235e066968bde09477b00944dbe5"
server_key_iv: "8bddc5f26feae2299699d0fc1dc8b295"
server_key_version: "encryptionkey@0"
I20260927 21:32:56.425787 4198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:56.447253 4198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:56.450656 4198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:56.450906 4198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:56.451836 4198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:32:56.453148 4198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:32:56.453217 4198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:56.453325 4198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:32:56.453385 4198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:56.485689 4198 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:35241
I20260927 21:32:56.485762 4325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:35241 every 8 connection(s)
I20260927 21:32:56.486827 4198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:32:56.494078 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4198
I20260927 21:32:56.494266 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:32:56.494668 3819 external_mini_cluster.cc:1428] Setting key 608309742c434297cabe5d9a236ef1cf
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:32:56.505224 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.489398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:51475 (local address 127.3.186.254:39495)
0927 21:32:56.489594 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.489598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.490611 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 21:32:56.492303 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.492309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.492402 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.492509 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.493884 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.494691 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.496513 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.496926 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.497065 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.500327 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.500332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.500335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.500364 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.502968 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.503568 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.503571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.503574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.503694 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.504035 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.504038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.504040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.504258 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.504367 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.504889 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.505028 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:32:56.506071 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.489821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:39495 (local address 127.3.186.195:51475)
0927 21:32:56.490454 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:56.490493 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:56.492049 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:56.492720 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:56.492727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:56.493170 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:56.493726 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:56.493743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.494864 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.494870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:56.496368 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:56.496382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.497166 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.497173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:56.497465 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.498191 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:56.498211 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:56.500166 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:56.503086 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:56.503092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:56.503101 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:56.503472 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:56.503796 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:56.503799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:56.503801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:56.503945 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:56.504357 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:56.504362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:56.504613 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 21:32:56.504806 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 21:32:56.505091 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":240,"threads_started":1}
I20260927 21:32:56.507215 4326 heartbeater.cc:347] Connected to a master server at 127.3.186.254:39495
I20260927 21:32:56.507431 4326 heartbeater.cc:464] Registering TS with master...
I20260927 21:32:56.508344 4326 heartbeater.cc:511] Master 127.3.186.254:39495 requested a full tablet report, sending...
I20260927 21:32:56.509598 3871 ts_manager.cc:194] Registered new tserver with Master: 63822de4be414026a663b71cfa92b09e (127.3.186.195:35241)
I20260927 21:32:56.510437 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:32:56.510569 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:51475
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:32:56.551716 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.537179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.3.186.193:38529)
0927 21:32:56.537784 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.537795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.537924 (+ 129us) server_negotiation.cc:407] Connection header received
0927 21:32:56.538136 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.538143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.538324 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.538490 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.539790 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.540722 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.541973 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.542512 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.542658 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.546545 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.546554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.546565 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.546592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.548901 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.549483 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.549488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.549490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.549665 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.550117 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.550121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.550123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.550450 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.550708 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.551104 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.551341 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":337,"threads_started":1}
W20260927 21:32:56.553013 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40156
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260927 21:32:56.581184 4060 heartbeater.cc:503] Master 127.3.186.254:39495 was elected leader, sending a full tablet report...
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:32:56.592192 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.579497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.3.186.193:38529)
0927 21:32:56.579717 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.579727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.579940 (+ 213us) server_negotiation.cc:407] Connection header received
0927 21:32:56.580123 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.580129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.580202 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.580299 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.581768 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.582700 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.583933 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.584319 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.584406 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.587677 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.587682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.587685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.587711 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.590380 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.590904 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.590907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.590909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.591003 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.591269 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.591272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.591274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.591490 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.591637 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.591855 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.591987 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 21:32:56.593549 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 3932
I20260927 21:32:56.605618 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.594336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.3.186.193:38529)
0927 21:32:56.594497 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.594505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.594771 (+ 266us) server_negotiation.cc:407] Connection header received
0927 21:32:56.594933 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.594938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.594997 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.595086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.596070 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.596687 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.598181 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.598628 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.598783 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.600964 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.600969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.600974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.601012 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.603239 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.603846 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.603851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.603854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.603966 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.604303 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.604310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.604314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.604567 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.604773 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.605046 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.605221 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:32:56.606221 4055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:56.686445 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 3932
I20260927 21:32:56.701972 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4065
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:32:56.716099 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.702784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.3.186.194:46539)
0927 21:32:56.703299 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.703306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.703331 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:32:56.703386 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.703392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.703591 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.703709 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.704806 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.705849 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.706961 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.707464 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.707552 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.711288 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.711297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.711311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.711337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.713825 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.714367 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.714372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.714375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.714481 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.714763 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.714765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.714767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.715057 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.715257 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.715601 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.715798 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":268,"threads_started":1}
I20260927 21:32:56.717334 4188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:56.796118 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4065
I20260927 21:32:56.817395 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4198
Sep 27 21:32:56 dist-test-slave-pqr8 krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:32:56.832662 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:56.818275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.3.186.195:35241)
0927 21:32:56.818850 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:56.818857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:56.818883 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:32:56.818937 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:56.818953 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:56.819171 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:56.819289 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:56.820466 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.821445 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.822866 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:56.823390 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:56.823485 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:56.827318 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:56.827329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:56.827341 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:56.827373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:56.830367 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.830920 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.830927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.830929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.831043 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:56.831321 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:56.831324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:56.831326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:56.831652 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:56.831861 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:56.832087 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 21:32:56.832292 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":313,"threads_started":1}
I20260927 21:32:56.834004 4321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:56.913259 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4198
I20260927 21:32:56.928895 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 3839
I20260927 21:32:56.930033 3901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:57.025493 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 3839
2026-09-27T21:32:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3667 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:57 dist-test-slave-pqr8 krb5kdc[4347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:57 dist-test-slave-pqr8 krb5kdc[4347](info): set up 2 sockets
Sep 27 21:32:57 dist-test-slave-pqr8 krb5kdc[4347](info): commencing operation
krb5kdc: starting...
W20260927 21:32:58.068960 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:32:58 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:32:58.220758 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41559
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:58.507663 4360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:58.507894 4360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:58.507978 4360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:58.514547 4360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:58.514627 4360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:58.514668 4360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:58.514719 4360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:58.514753 4360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:58.523309 4360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:41559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:58.524300 4360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:58.525352 4360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:58.536012 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:58.536060 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:58.536099 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:58.536844 4360 server_base.cc:1034] running on GCE node
I20260927 21:32:58.537569 4360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:58.538751 4360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:58.539961 4360 hybrid_clock.cc:648] HybridClock initialized: now 1790544778539924 us; error 50 us; skew 500 ppm
I20260927 21:32:58.540273 4360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:58 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:58.546865 4360 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:32:58.548095 4360 webserver.cc:460] Webserver started at http://127.3.186.254:46193/ using document root <none> and password file <none>
I20260927 21:32:58.548616 4360 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:58.548698 4360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:58.548907 4360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:58.551548 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "3e71bf0b97b64c629782ea317f27cd62"
format_stamp: "Formatted at 2026-09-27 21:32:58 on dist-test-slave-pqr8"
server_key: "a3064c35457904acedfb6bc08f41a597"
server_key_iv: "4ab76c711d45251d7f80bc58151f88ef"
server_key_version: "encryptionkey@0"
I20260927 21:32:58.552213 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "3e71bf0b97b64c629782ea317f27cd62"
format_stamp: "Formatted at 2026-09-27 21:32:58 on dist-test-slave-pqr8"
server_key: "a3064c35457904acedfb6bc08f41a597"
server_key_iv: "4ab76c711d45251d7f80bc58151f88ef"
server_key_version: "encryptionkey@0"
I20260927 21:32:58.556700 4360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:32:58.559641 4375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:58.560817 4360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:32:58.561002 4360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "3e71bf0b97b64c629782ea317f27cd62"
format_stamp: "Formatted at 2026-09-27 21:32:58 on dist-test-slave-pqr8"
server_key: "a3064c35457904acedfb6bc08f41a597"
server_key_iv: "4ab76c711d45251d7f80bc58151f88ef"
server_key_version: "encryptionkey@0"
I20260927 21:32:58.561177 4360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:58.578707 4360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:58.582271 4360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:58.582621 4360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:58.599895 4360 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:41559
I20260927 21:32:58.599926 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:41559 every 8 connection(s)
I20260927 21:32:58.601117 4360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:58.602877 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4360
I20260927 21:32:58.603103 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:32:58.603864 3819 external_mini_cluster.cc:1428] Setting key 892c661f6f532e86c7d141eaa56b8fbd
I20260927 21:32:58.605168 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:58.615342 4427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62: Bootstrap starting.
Sep 27 21:32:58 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:32:58.618496 4427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:58.619401 4427 log.cc:824] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:58.621905 4427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62: No bootstrap required, opened a new log
I20260927 21:32:58.623461 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:58.605706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.3.186.254:41559)
0927 21:32:58.606334 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:58.606344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:58.606377 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:32:58.607783 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:58.607801 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:58.608146 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:58.608516 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:58.609860 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:58.611099 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:58.612263 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:58.612924 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:58.613154 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:58.617132 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:58.617139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:58.617150 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:58.617178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:58.619869 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:58.620468 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:58.620474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:58.620479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:58.620659 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:58.621028 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:58.621031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:58.621033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:58.621435 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:58.621768 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:58.622197 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 21:32:58.622568 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":277,"threads_started":1}
I20260927 21:32:58.629069 4427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e71bf0b97b64c629782ea317f27cd62" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41559 } }
I20260927 21:32:58.629405 4427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:58.629513 4427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e71bf0b97b64c629782ea317f27cd62, State: Initialized, Role: FOLLOWER
I20260927 21:32:58.630029 4427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e71bf0b97b64c629782ea317f27cd62" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41559 } }
I20260927 21:32:58.630232 4427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:58.630366 4427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:58.630553 4427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:58.632678 4427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e71bf0b97b64c629782ea317f27cd62" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41559 } }
I20260927 21:32:58.633052 4427 leader_election.cc:302] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e71bf0b97b64c629782ea317f27cd62; no voters:
I20260927 21:32:58.633572 4427 leader_election.cc:288] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:58.633692 4432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:58.634750 4432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [term 1 LEADER]: Becoming Leader. State: Replica: 3e71bf0b97b64c629782ea317f27cd62, State: Running, Role: LEADER
I20260927 21:32:58.635221 4432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e71bf0b97b64c629782ea317f27cd62" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41559 } }
I20260927 21:32:58.635448 4427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:58.638365 4434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e71bf0b97b64c629782ea317f27cd62. Latest consensus state: current_term: 1 leader_uuid: "3e71bf0b97b64c629782ea317f27cd62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e71bf0b97b64c629782ea317f27cd62" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41559 } } }
I20260927 21:32:58.638324 4433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e71bf0b97b64c629782ea317f27cd62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e71bf0b97b64c629782ea317f27cd62" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41559 } } }
I20260927 21:32:58.638675 4433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:58.638679 4434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:58.641791 4441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:58.645653 4441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:58.652303 4441 catalog_manager.cc:1347] Generated new cluster ID: 868f2e4fce7941739b16012a6143a43c
I20260927 21:32:58.652393 4441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:58.678668 4441 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:58.679759 4441 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:58.686306 4441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e71bf0b97b64c629782ea317f27cd62: Generated new TSK 0
I20260927 21:32:58.687150 4441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:32:58.730989 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41559
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:59.029824 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:59.030071 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:59.030171 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:59.037062 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:59.037147 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:59.037246 4453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:32:59.046305 4453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:59.047345 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:59.048489 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:59.060837 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:59.060745 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:59.060719 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:59.061312 4453 server_base.cc:1034] running on GCE node
I20260927 21:32:59.061995 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:59.062870 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:59.064088 4453 hybrid_clock.cc:648] HybridClock initialized: now 1790544779064049 us; error 53 us; skew 500 ppm
I20260927 21:32:59.064388 4453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:59 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:59.067502 4453 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:32:59.068722 4453 webserver.cc:460] Webserver started at http://127.3.186.193:39071/ using document root <none> and password file <none>
I20260927 21:32:59.069324 4453 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:59.069478 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:59.069788 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:59.072466 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "6a285e74c6d7419cb9517a9d0c872c23"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "d5caf0c4657589f39af6965324a3cd87"
server_key_iv: "5d1c1a85206b1df8cc8eea3b77dae17d"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.073206 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "6a285e74c6d7419cb9517a9d0c872c23"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "d5caf0c4657589f39af6965324a3cd87"
server_key_iv: "5d1c1a85206b1df8cc8eea3b77dae17d"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.077708 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 21:32:59.080765 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.081995 4453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:32:59.082157 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "6a285e74c6d7419cb9517a9d0c872c23"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "d5caf0c4657589f39af6965324a3cd87"
server_key_iv: "5d1c1a85206b1df8cc8eea3b77dae17d"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.082340 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:59.098453 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:59.102401 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:59.102648 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:59.103417 4453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:32:59.104521 4453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:32:59.104591 4453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.104668 4453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:32:59.104709 4453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.135746 4453 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:39285
I20260927 21:32:59.135785 4580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:39285 every 8 connection(s)
I20260927 21:32:59.137023 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:32:59.143963 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4453
I20260927 21:32:59.144201 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:32:59.144905 3819 external_mini_cluster.cc:1428] Setting key ffe0daee4f5fa3d9b0dcbc790e89e7ad
Sep 27 21:32:59 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:32:59.156978 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:59.139623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:60483 (local address 127.3.186.254:41559)
0927 21:32:59.140061 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:59.140065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:59.140957 (+ 892us) server_negotiation.cc:407] Connection header received
0927 21:32:59.142692 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:59.142696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:59.142757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:59.142856 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:59.144249 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.145331 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.147053 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.147709 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.147852 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:59.151483 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:59.151487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:59.151491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:59.151518 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:59.154039 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:59.154910 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:59.154914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:59.154916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:59.155017 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:59.155457 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:59.155460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:59.155462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:59.155700 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:59.155797 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:59.156594 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 21:32:59.156734 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260927 21:32:59.158353 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:59.140176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41559 (local address 127.3.186.193:60483)
0927 21:32:59.140811 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:59.140847 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:59.142445 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:59.143047 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:59.143054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:59.143511 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:59.144050 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:59.144065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.145567 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.145572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:59.146921 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:59.146931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.147901 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.147905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:59.148096 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:59.148745 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:59.148801 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:59.151259 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:59.154172 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:59.154182 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:59.154199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:59.154793 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:59.155113 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:59.155120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:59.155126 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:59.155345 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:59.155821 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:59.155831 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:59.156219 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 21:32:59.156495 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 21:32:59.157005 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":240,"threads_started":1}
I20260927 21:32:59.159755 4581 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41559
I20260927 21:32:59.160041 4581 heartbeater.cc:464] Registering TS with master...
I20260927 21:32:59.160876 4581 heartbeater.cc:511] Master 127.3.186.254:41559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
I20260927 21:32:59.162988 4392 ts_manager.cc:194] Registered new tserver with Master: 6a285e74c6d7419cb9517a9d0c872c23 (127.3.186.193:39285)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
I20260927 21:32:59.164746 4392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:60483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:32:59.184042 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41559
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:59.477928 4587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:59.478209 4587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:59.478384 4587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:59.484938 4587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:59.485083 4587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:59.485311 4587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:32:59.494603 4587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:59.495703 4587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:59.496925 4587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:59.508994 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:59.509049 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:59.509094 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:59.509755 4587 server_base.cc:1034] running on GCE node
I20260927 21:32:59.510231 4587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:59.511000 4587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:59.512221 4587 hybrid_clock.cc:648] HybridClock initialized: now 1790544779512182 us; error 50 us; skew 500 ppm
I20260927 21:32:59.512501 4587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:59 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:59.515421 4587 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:32:59.516726 4587 webserver.cc:460] Webserver started at http://127.3.186.194:37411/ using document root <none> and password file <none>
I20260927 21:32:59.517323 4587 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:59.517457 4587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:59.517767 4587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:59.520319 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "72f8f9a868234e0e829b5da3adbc34d2"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "29237d5b493e852aab209dd46c0cd7e0"
server_key_iv: "7303d153c3ed049cbf67ca4e4cb8ba2b"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.521035 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "72f8f9a868234e0e829b5da3adbc34d2"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "29237d5b493e852aab209dd46c0cd7e0"
server_key_iv: "7303d153c3ed049cbf67ca4e4cb8ba2b"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.525971 4587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:32:59.529031 4602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.530277 4587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:32:59.530438 4587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "72f8f9a868234e0e829b5da3adbc34d2"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "29237d5b493e852aab209dd46c0cd7e0"
server_key_iv: "7303d153c3ed049cbf67ca4e4cb8ba2b"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.530635 4587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:59.542775 4587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:59.546129 4587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:59.546404 4587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:59.547222 4587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:32:59.548353 4587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:32:59.548460 4587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.548600 4587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:32:59.548668 4587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.579219 4587 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:44811
I20260927 21:32:59.579267 4714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:44811 every 8 connection(s)
I20260927 21:32:59.580341 4587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:32:59.586676 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4587
I20260927 21:32:59.586860 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:32:59.587306 3819 external_mini_cluster.cc:1428] Setting key 030957716314af00810ab7fe4626fdca
Sep 27 21:32:59 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:32:59.601565 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:59.582643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:43527 (local address 127.3.186.254:41559)
0927 21:32:59.582810 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:59.582815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:59.583909 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 21:32:59.585804 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:59.585809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:59.585916 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:59.586023 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:59.587787 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.588636 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.590804 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.591206 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.591300 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:59.595555 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:59.595559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:59.595562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:59.595589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:59.597952 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:59.598791 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:59.598795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:59.598797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:59.598903 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:59.599373 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:59.599398 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:59.599402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:59.599734 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:59.599877 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:59.600776 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 21:32:59.601033 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260927 21:32:59.602306 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:59.583110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41559 (local address 127.3.186.194:43527)
0927 21:32:59.583767 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:59.583801 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:59.585501 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:59.586283 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:59.586294 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:59.587003 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:59.587638 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:59.587656 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.588771 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.588776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:59.590649 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:59.590665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:59.591452 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:59.591461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:59.591829 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:59.592490 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:59.592522 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:59.595385 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:59.598072 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:59.598082 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:59.598106 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:59.598684 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:59.599007 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:59.599014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:59.599021 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:59.599246 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:59.599950 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:59.599961 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:59.600356 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 21:32:59.600604 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 21:32:59.601079 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:32:59.605082 4715 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41559
I20260927 21:32:59.605371 4715 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
I20260927 21:32:59.606137 4715 heartbeater.cc:511] Master 127.3.186.254:41559 requested a full tablet report, sending...
I20260927 21:32:59.607513 4392 ts_manager.cc:194] Registered new tserver with Master: 72f8f9a868234e0e829b5da3adbc34d2 (127.3.186.194:44811)
I20260927 21:32:59.608345 4392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:43527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:32:59.626133 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41559
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:59.921895 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:59.922238 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:59.922478 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:59.929340 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:59.929476 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:59.929666 4720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:32:59.938897 4720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:59.939997 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:59.941218 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:59.953469 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:59.953436 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:59.953635 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:59.954202 4720 server_base.cc:1034] running on GCE node
I20260927 21:32:59.954694 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:59.955417 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:59.956643 4720 hybrid_clock.cc:648] HybridClock initialized: now 1790544779956593 us; error 40 us; skew 500 ppm
I20260927 21:32:59.956944 4720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:59 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:59.959795 4720 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:32:59.961114 4720 webserver.cc:460] Webserver started at http://127.3.186.195:40017/ using document root <none> and password file <none>
I20260927 21:32:59.961845 4720 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:59.961977 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:59.962252 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:59.964880 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "cba04ffe347745d3a8cae1f92f7dfcab"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "e014c7ba63d94c023b13032a9f20a3e4"
server_key_iv: "0b8769a5f8143a1e545d05f3d03c6ee4"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.965679 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "cba04ffe347745d3a8cae1f92f7dfcab"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "e014c7ba63d94c023b13032a9f20a3e4"
server_key_iv: "0b8769a5f8143a1e545d05f3d03c6ee4"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.970103 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:32:59.973083 4735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:59.974277 4720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:32:59.974442 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "cba04ffe347745d3a8cae1f92f7dfcab"
format_stamp: "Formatted at 2026-09-27 21:32:59 on dist-test-slave-pqr8"
server_key: "e014c7ba63d94c023b13032a9f20a3e4"
server_key_iv: "0b8769a5f8143a1e545d05f3d03c6ee4"
server_key_version: "encryptionkey@0"
I20260927 21:32:59.974637 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:00.003207 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:00.006781 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:00.007048 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:00.007954 4720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:00.009164 4720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:00.009262 4720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:00.009404 4720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:00.009480 4720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:00.040524 4720 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:45157
I20260927 21:33:00.040575 4847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:45157 every 8 connection(s)
I20260927 21:33:00.041797 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:00.048878 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4720
I20260927 21:33:00.049117 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:00.049577 3819 external_mini_cluster.cc:1428] Setting key ca3eed9049f3662811392900b50a89ce
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:00.060245 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.044339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:32939 (local address 127.3.186.254:41559)
0927 21:33:00.044510 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.044515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.045654 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 21:33:00.047311 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.047316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.047398 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.047504 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.049096 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.049938 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.051620 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.052176 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.052301 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.055490 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.055494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.055497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.055523 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.057885 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.058542 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.058546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.058548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.058658 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.058999 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.059002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.059004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.059225 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.059324 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.059885 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.060015 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:33:00.061064 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.044815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41559 (local address 127.3.186.195:32939)
0927 21:33:00.045458 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:00.045493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:00.047058 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:00.047732 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:00.047739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:00.048191 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:00.048843 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.048869 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.050102 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.050107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.051491 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.051502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.052387 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.052394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.052641 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.053220 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:00.053239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:00.055322 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:00.058002 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:00.058007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:00.058017 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:00.058444 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:00.058750 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:00.058753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:00.058755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:00.058911 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:00.059322 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:00.059326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:00.059589 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 21:33:00.059811 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 21:33:00.060122 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
I20260927 21:33:00.062325 4848 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41559
I20260927 21:33:00.062640 4848 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:00.063391 4848 heartbeater.cc:511] Master 127.3.186.254:41559 requested a full tablet report, sending...
I20260927 21:33:00.064888 4392 ts_manager.cc:194] Registered new tserver with Master: cba04ffe347745d3a8cae1f92f7dfcab (127.3.186.195:45157)
I20260927 21:33:00.065821 4392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:32939
I20260927 21:33:00.076529 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:33:00.093938 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.082303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.3.186.254:41559)
0927 21:33:00.082444 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.082448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.082992 (+ 544us) server_negotiation.cc:407] Connection header received
0927 21:33:00.083116 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.083120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.083171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.083241 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.084931 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.085723 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.087088 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.087492 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.087595 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.089617 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.089621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.089624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.089650 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.091741 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.092321 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.092324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.092326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.092427 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.092711 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.092713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.092715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.092983 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.093145 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.093408 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.093613 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:33:00.099349 4392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:00.106961 4392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:00.132370 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.117134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:44811 (local address 127.0.0.1:53366)
0927 21:33:00.122458 (+ 5324us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:00.122499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:00.122723 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:00.124045 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:00.124056 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:00.124107 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:00.124691 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.124707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.127265 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.127273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.129681 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.129693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.131382 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.131387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.131577 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.131619 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:33:00.131742 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 21:33:00.131915 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5035,"mutex_wait_us":136,"thread_start_us":7786,"threads_started":1}
I20260927 21:33:00.132817 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.117633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.3.186.194:44811)
0927 21:33:00.123355 (+ 5722us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.123367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.123398 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:33:00.123506 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.123516 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.123782 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.124921 (+ 1139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:00.125175 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.127085 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.129843 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.130969 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.131100 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.131271 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.132108 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.132266 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5562,"thread_start_us":5451,"threads_started":1}
I20260927 21:33:00.133651 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.117270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39285 (local address 127.0.0.1:51764)
0927 21:33:00.122891 (+ 5621us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:00.122909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:00.123065 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:00.124111 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:00.124119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:00.124165 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:00.124806 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.124821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.127306 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.127314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.130709 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.130718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.131876 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.131882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.132058 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.132086 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:33:00.132174 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:33:00.132299 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5509,"mutex_wait_us":123,"thread_start_us":4734,"threads_started":1}
I20260927 21:33:00.134081 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.117529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.3.186.193:39285)
0927 21:33:00.118563 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.118575 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.123193 (+ 4618us) server_negotiation.cc:407] Connection header received
0927 21:33:00.123301 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.123310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.123584 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.123812 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:00.125063 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.126673 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.130837 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.131676 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.131824 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.131997 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.132870 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.133040 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":250,"threads_started":1}
I20260927 21:33:00.136628 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.122858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.3.186.195:45157)
0927 21:33:00.128569 (+ 5711us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.128577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.129254 (+ 677us) server_negotiation.cc:407] Connection header received
0927 21:33:00.129354 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.129362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.129593 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.129766 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:00.130528 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.131943 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.134270 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.135565 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.135698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.135887 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.136187 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.136326 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5522,"thread_start_us":5351,"threads_started":1}
I20260927 21:33:00.137002 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.122292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:45157 (local address 127.0.0.1:51746)
0927 21:33:00.128999 (+ 6707us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:00.129015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:00.129134 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:00.129980 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:00.129984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:00.130005 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:00.130352 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.130361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.132108 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.132114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.134084 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.134101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.135777 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.135787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.135954 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.135988 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:33:00.136145 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 21:33:00.136267 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6587,"thread_start_us":6453,"threads_started":1}
I20260927 21:33:00.136762 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b8a84531f03a44dea302bc4ef61b1e6d (DEFAULT_TABLE table=test-table [id=6c17479699f94fe0b115c5d8cd9a2ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:00.138265 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a84531f03a44dea302bc4ef61b1e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:00.140097 4516 tablet_service.cc:1467] Processing CreateTablet for tablet b8a84531f03a44dea302bc4ef61b1e6d (DEFAULT_TABLE table=test-table [id=6c17479699f94fe0b115c5d8cd9a2ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:00.141306 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a84531f03a44dea302bc4ef61b1e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:00.145370 4783 tablet_service.cc:1467] Processing CreateTablet for tablet b8a84531f03a44dea302bc4ef61b1e6d (DEFAULT_TABLE table=test-table [id=6c17479699f94fe0b115c5d8cd9a2ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:00.146976 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a84531f03a44dea302bc4ef61b1e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:00.149863 4866 tablet_bootstrap.cc:492] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: Bootstrap starting.
I20260927 21:33:00.153354 4866 tablet_bootstrap.cc:654] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:00.154706 4867 tablet_bootstrap.cc:492] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: Bootstrap starting.
I20260927 21:33:00.155022 4866 log.cc:824] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:00.157812 4867 tablet_bootstrap.cc:654] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:00.158010 4866 tablet_bootstrap.cc:492] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: No bootstrap required, opened a new log
I20260927 21:33:00.158267 4866 ts_tablet_manager.cc:1397] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260927 21:33:00.158918 4867 log.cc:824] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:00.165180 4869 tablet_bootstrap.cc:492] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: Bootstrap starting.
I20260927 21:33:00.166098 4867 tablet_bootstrap.cc:492] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: No bootstrap required, opened a new log
I20260927 21:33:00.166332 4867 ts_tablet_manager.cc:1397] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260927 21:33:00.167692 4869 tablet_bootstrap.cc:654] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:00.167891 4581 heartbeater.cc:503] Master 127.3.186.254:41559 was elected leader, sending a full tablet report...
I20260927 21:33:00.168787 4869 log.cc:824] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:00.170750 4866 raft_consensus.cc:348] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.171257 4866 raft_consensus.cc:374] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:00.171379 4866 raft_consensus.cc:729] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f8f9a868234e0e829b5da3adbc34d2, State: Initialized, Role: FOLLOWER
I20260927 21:33:00.172153 4866 consensus_queue.cc:260] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.176249 4715 heartbeater.cc:503] Master 127.3.186.254:41559 was elected leader, sending a full tablet report...
I20260927 21:33:00.177145 4866 ts_tablet_manager.cc:1428] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: Time spent starting tablet: real 0.019s user 0.011s sys 0.006s
I20260927 21:33:00.177376 4867 raft_consensus.cc:348] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.177882 4867 raft_consensus.cc:374] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:00.178027 4867 raft_consensus.cc:729] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a285e74c6d7419cb9517a9d0c872c23, State: Initialized, Role: FOLLOWER
I20260927 21:33:00.178704 4867 consensus_queue.cc:260] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.179809 4869 tablet_bootstrap.cc:492] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: No bootstrap required, opened a new log
I20260927 21:33:00.180076 4869 ts_tablet_manager.cc:1397] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260927 21:33:00.180939 4867 ts_tablet_manager.cc:1428] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 21:33:00.186718 4869 raft_consensus.cc:348] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.186998 4869 raft_consensus.cc:374] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:00.187083 4869 raft_consensus.cc:729] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba04ffe347745d3a8cae1f92f7dfcab, State: Initialized, Role: FOLLOWER
I20260927 21:33:00.187536 4869 consensus_queue.cc:260] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.188330 4848 heartbeater.cc:503] Master 127.3.186.254:41559 was elected leader, sending a full tablet report...
I20260927 21:33:00.188717 4869 ts_tablet_manager.cc:1428] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260927 21:33:00.244684 4872 raft_consensus.cc:482] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:00.245007 4872 raft_consensus.cc:504] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.246814 4872 leader_election.cc:288] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a285e74c6d7419cb9517a9d0c872c23 (127.3.186.193:39285), cba04ffe347745d3a8cae1f92f7dfcab (127.3.186.195:45157)
I20260927 21:33:00.253741 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.246965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39285 (local address 127.3.186.194:49307)
0927 21:33:00.247806 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:00.247822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:00.247991 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:00.248354 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:00.248360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:00.248385 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:00.248731 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.248743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.249839 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.249843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.251818 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.251831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.253128 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.253132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.253240 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.253263 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:33:00.253333 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:33:00.253413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":341,"threads_started":1}
I20260927 21:33:00.253777 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.247170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:49307 (local address 127.3.186.193:39285)
0927 21:33:00.247361 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.247370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.247922 (+ 552us) server_negotiation.cc:407] Connection header received
0927 21:33:00.248160 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.248164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.248220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.248318 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:00.248882 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.249717 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.251961 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.252945 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.253102 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.253171 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.253382 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.253500 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:33:00.254630 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.247506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:48311 (local address 127.3.186.195:45157)
0927 21:33:00.247701 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.247708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.248291 (+ 583us) server_negotiation.cc:407] Connection header received
0927 21:33:00.248474 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.248479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.248544 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.248615 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:00.249646 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.250368 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.252427 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.253642 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.253789 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.253863 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.254286 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.254404 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:33:00.254657 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.247282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:45157 (local address 127.3.186.194:48311)
0927 21:33:00.248196 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:00.248209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:00.248339 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:00.248693 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:00.248699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:00.248726 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:00.249465 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.249474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.250503 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.250508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.252292 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:00.252301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.253856 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.253865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:00.254065 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.254097 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:33:00.254233 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:33:00.254358 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":612,"threads_started":1}
I20260927 21:33:00.254863 4536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a84531f03a44dea302bc4ef61b1e6d" candidate_uuid: "72f8f9a868234e0e829b5da3adbc34d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a285e74c6d7419cb9517a9d0c872c23" is_pre_election: true
I20260927 21:33:00.255358 4536 raft_consensus.cc:2454] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f8f9a868234e0e829b5da3adbc34d2 in term 0.
I20260927 21:33:00.255540 4803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a84531f03a44dea302bc4ef61b1e6d" candidate_uuid: "72f8f9a868234e0e829b5da3adbc34d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" is_pre_election: true
I20260927 21:33:00.255970 4803 raft_consensus.cc:2454] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f8f9a868234e0e829b5da3adbc34d2 in term 0.
I20260927 21:33:00.255975 4604 leader_election.cc:302] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a285e74c6d7419cb9517a9d0c872c23, 72f8f9a868234e0e829b5da3adbc34d2; no voters:
I20260927 21:33:00.256326 4872 raft_consensus.cc:2781] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:00.256501 4872 raft_consensus.cc:482] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:00.256632 4872 raft_consensus.cc:3037] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:00.258749 4872 raft_consensus.cc:504] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.259343 4872 leader_election.cc:288] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [CANDIDATE]: Term 1 election: Requested vote from peers 6a285e74c6d7419cb9517a9d0c872c23 (127.3.186.193:39285), cba04ffe347745d3a8cae1f92f7dfcab (127.3.186.195:45157)
I20260927 21:33:00.259634 4536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a84531f03a44dea302bc4ef61b1e6d" candidate_uuid: "72f8f9a868234e0e829b5da3adbc34d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a285e74c6d7419cb9517a9d0c872c23"
I20260927 21:33:00.259804 4536 raft_consensus.cc:3037] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:00.259793 4803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a84531f03a44dea302bc4ef61b1e6d" candidate_uuid: "72f8f9a868234e0e829b5da3adbc34d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba04ffe347745d3a8cae1f92f7dfcab"
I20260927 21:33:00.259963 4803 raft_consensus.cc:3037] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:00.262291 4536 raft_consensus.cc:2454] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f8f9a868234e0e829b5da3adbc34d2 in term 1.
I20260927 21:33:00.262494 4803 raft_consensus.cc:2454] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f8f9a868234e0e829b5da3adbc34d2 in term 1.
I20260927 21:33:00.262670 4604 leader_election.cc:302] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a285e74c6d7419cb9517a9d0c872c23, 72f8f9a868234e0e829b5da3adbc34d2; no voters:
I20260927 21:33:00.262928 4872 raft_consensus.cc:2781] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:00.263310 4872 raft_consensus.cc:686] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [term 1 LEADER]: Becoming Leader. State: Replica: 72f8f9a868234e0e829b5da3adbc34d2, State: Running, Role: LEADER
I20260927 21:33:00.263754 4872 consensus_queue.cc:237] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } }
I20260927 21:33:00.267881 4391 catalog_manager.cc:5495] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72f8f9a868234e0e829b5da3adbc34d2 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "72f8f9a868234e0e829b5da3adbc34d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f8f9a868234e0e829b5da3adbc34d2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a285e74c6d7419cb9517a9d0c872c23" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cba04ffe347745d3a8cae1f92f7dfcab" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 45157 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:00.290490 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.276481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.3.186.193:39285)
0927 21:33:00.276622 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.276626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.277120 (+ 494us) server_negotiation.cc:407] Connection header received
0927 21:33:00.277317 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.277320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.277379 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.277475 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.278965 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.280062 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.281281 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.281738 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.281868 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.285618 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.285629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.285641 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.285687 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.288312 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.288841 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.288848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.288850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.288974 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.289257 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.289260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.289262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.289614 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.289822 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.290029 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.290158 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 21:33:00.293334 4849 tablet.cc:2369] T b8a84531f03a44dea302bc4ef61b1e6d P cba04ffe347745d3a8cae1f92f7dfcab: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:00.332226 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.319909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.3.186.193:39285)
0927 21:33:00.320114 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.320119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.320226 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:33:00.320347 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.320350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.320403 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.320478 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.322105 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.322807 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.323954 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.324313 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.324396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.327736 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.327741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.327744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.327770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.330193 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.330708 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.330711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.330713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.330816 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.331135 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.331137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.331140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.331350 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.331512 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.331746 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.331878 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 21:33:00.332264 4716 tablet.cc:2369] T b8a84531f03a44dea302bc4ef61b1e6d P 72f8f9a868234e0e829b5da3adbc34d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:00.371716 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.358781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.3.186.193:39285)
0927 21:33:00.358979 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.358987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.359240 (+ 253us) server_negotiation.cc:407] Connection header received
0927 21:33:00.359420 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.359425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.359484 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.359575 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.360824 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.361452 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.362650 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.363058 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.363195 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.367229 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.367233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.367237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.367263 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.369400 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.370256 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.370259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.370262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.370373 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.370685 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.370688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.370691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.370940 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.371108 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.371324 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.371469 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 21:33:00.373256 4496 tablet_service.cc:2908] Rejecting scan request for tablet b8a84531f03a44dea302bc4ef61b1e6d: Uninitialized: clean time has not yet been initialized
I20260927 21:33:00.376226 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4453
I20260927 21:33:00.387797 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.377030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.3.186.193:39285)
0927 21:33:00.377193 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.377200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.377492 (+ 292us) server_negotiation.cc:407] Connection header received
0927 21:33:00.377721 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.377732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.377790 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.377884 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.378904 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.379508 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.380605 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.380981 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.381128 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.383125 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.383131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.383135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.383172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.385407 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.386026 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.386032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.386037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.386166 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.386518 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.386523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.386526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.386813 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.387108 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.387283 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.387479 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:33:00.388326 4576 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 21:33:00.473033 4582 tablet.cc:2369] T b8a84531f03a44dea302bc4ef61b1e6d P 6a285e74c6d7419cb9517a9d0c872c23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:00.473778 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4453
I20260927 21:33:00.491207 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4587
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:00.504716 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.491906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.3.186.194:44811)
0927 21:33:00.492095 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.492102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.492121 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:33:00.492311 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.492315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.492372 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.492472 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.493615 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.494338 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.495559 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.495947 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.496071 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.499753 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.499765 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.499779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.499812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.502224 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.502831 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.502839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.502842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.502965 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.503340 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.503346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.503349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.503669 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.503947 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.504236 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.504450 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:33:00.505398 4710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:00.591162 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4587
I20260927 21:33:00.607052 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4720
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:00.620340 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:00.607797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.3.186.195:45157)
0927 21:33:00.607977 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:00.607982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:00.608001 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:33:00.608148 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:00.608152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:00.608210 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:00.608312 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:00.609443 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.610196 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.611373 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:00.611736 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:00.611858 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:00.615560 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:00.615571 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:00.615585 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:00.615612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:00.617956 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.618546 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.618553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.618555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.618665 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:00.618991 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:00.618996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:00.618999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:00.619301 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:00.619625 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:00.619856 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 21:33:00.620051 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:33:00.620919 4843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:00.703460 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4720
I20260927 21:33:00.718842 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4360
I20260927 21:33:00.719884 4422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:00.820034 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4360
2026-09-27T21:33:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3795 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4896](info): set up 2 sockets
Sep 27 21:33:00 dist-test-slave-pqr8 krb5kdc[4896](info): commencing operation
krb5kdc: starting...
W20260927 21:33:01.865415 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:01 dist-test-slave-pqr8 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:02.021178 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43809
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:43809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:02.315320 4909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:02.315552 4909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:02.315725 4909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:02.322189 4909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:02.322270 4909 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:33:02.322350 4909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:02.322446 4909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:02.322525 4909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:02.322610 4909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:02.331162 4909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:43809
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:02.332208 4909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:02.333576 4909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:02.344069 4914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:02.344117 4915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:02.344194 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:02.344882 4909 server_base.cc:1034] running on GCE node
I20260927 21:33:02.345486 4909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:02.346634 4909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:02.347862 4909 hybrid_clock.cc:648] HybridClock initialized: now 1790544782347802 us; error 37 us; skew 500 ppm
I20260927 21:33:02.348153 4909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:02 dist-test-slave-pqr8 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:02.350895 4909 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:02.352380 4909 webserver.cc:460] Webserver started at http://127.3.186.254:39247/ using document root <none> and password file <none>
I20260927 21:33:02.352989 4909 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:02.353108 4909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:02.353416 4909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:02.355960 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670"
format_stamp: "Formatted at 2026-09-27 21:33:02 on dist-test-slave-pqr8"
server_key: "216198aa2b3b2de1f5d0cd086399f6fd"
server_key_iv: "33680aed080193a7ae432858063d57cb"
server_key_version: "encryptionkey@0"
I20260927 21:33:02.356652 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670"
format_stamp: "Formatted at 2026-09-27 21:33:02 on dist-test-slave-pqr8"
server_key: "216198aa2b3b2de1f5d0cd086399f6fd"
server_key_iv: "33680aed080193a7ae432858063d57cb"
server_key_version: "encryptionkey@0"
I20260927 21:33:02.361055 4909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 21:33:02.364216 4924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:02.365315 4909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:02.365485 4909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670"
format_stamp: "Formatted at 2026-09-27 21:33:02 on dist-test-slave-pqr8"
server_key: "216198aa2b3b2de1f5d0cd086399f6fd"
server_key_iv: "33680aed080193a7ae432858063d57cb"
server_key_version: "encryptionkey@0"
I20260927 21:33:02.365731 4909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:02.392964 4909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:02.396507 4909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:02.396741 4909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:02.416378 4909 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:43809
I20260927 21:33:02.416409 4985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:43809 every 8 connection(s)
I20260927 21:33:02.417616 4909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:02.420820 4986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:02.423552 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 4909
I20260927 21:33:02.423785 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:02.424394 3819 external_mini_cluster.cc:1428] Setting key 0b4bb280011107cbdffae72249b3dcd7
I20260927 21:33:02.432019 4986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: Bootstrap starting.
I20260927 21:33:02.435470 4986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:02.436650 4986 log.cc:824] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:33:02 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:02.439131 4986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: No bootstrap required, opened a new log
I20260927 21:33:02.445282 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:02.426156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.3.186.254:43809)
0927 21:33:02.426844 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:02.426854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:02.426890 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:33:02.428411 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:02.428428 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:02.428756 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:02.429109 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:02.430663 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.431848 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.433596 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.434214 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.434335 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:02.438784 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:02.438791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:02.438803 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:02.438833 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:02.441689 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:02.442328 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:02.442334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:02.442339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:02.442480 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:02.442885 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:02.442889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:02.442891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:02.443351 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:02.443646 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:02.444056 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 21:33:02.444426 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":336,"threads_started":1}
I20260927 21:33:02.446491 4986 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43809 } }
I20260927 21:33:02.446892 4986 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:02.446988 4986 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfc7a5e3ad94a2f82bb02c68ee3e670, State: Initialized, Role: FOLLOWER
I20260927 21:33:02.447561 4986 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43809 } }
I20260927 21:33:02.447839 4986 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:02.448000 4986 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:02.448134 4986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:02.449980 4986 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43809 } }
I20260927 21:33:02.450351 4986 leader_election.cc:302] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfc7a5e3ad94a2f82bb02c68ee3e670; no voters:
I20260927 21:33:02.450837 4986 leader_election.cc:288] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:02.450948 4991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:02.451586 4991 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [term 1 LEADER]: Becoming Leader. State: Replica: ebfc7a5e3ad94a2f82bb02c68ee3e670, State: Running, Role: LEADER
I20260927 21:33:02.452080 4991 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43809 } }
I20260927 21:33:02.452576 4986 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:02.454511 4993 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfc7a5e3ad94a2f82bb02c68ee3e670. Latest consensus state: current_term: 1 leader_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43809 } } }
I20260927 21:33:02.454878 4993 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:02.454906 4992 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfc7a5e3ad94a2f82bb02c68ee3e670" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43809 } } }
I20260927 21:33:02.455219 4992 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:02.456897 4998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:02.461257 4998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:02.468868 4998 catalog_manager.cc:1347] Generated new cluster ID: 63756abf4a3d451386dae5b115b614b7
I20260927 21:33:02.468984 4998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:02.483779 4998 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:02.484903 4998 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:02.492813 4998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: Generated new TSK 0
I20260927 21:33:02.493450 4998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:02.534704 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43809
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:02.837841 5012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:02.838066 5012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:02.838224 5012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:02.844863 5012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:02.844949 5012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:02.845023 5012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:02.845180 5012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:02.854296 5012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:02.855336 5012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:02.856606 5012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:02.868680 5017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:02.868649 5018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:02.868727 5020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:02.869851 5012 server_base.cc:1034] running on GCE node
I20260927 21:33:02.870373 5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:02.871137 5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:02.872363 5012 hybrid_clock.cc:648] HybridClock initialized: now 1790544782872309 us; error 58 us; skew 500 ppm
I20260927 21:33:02.872629 5012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:02 dist-test-slave-pqr8 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:02.875672 5012 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:02.876866 5012 webserver.cc:460] Webserver started at http://127.3.186.193:33853/ using document root <none> and password file <none>
I20260927 21:33:02.877501 5012 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:02.877681 5012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:02.877986 5012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:02.880484 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e"
format_stamp: "Formatted at 2026-09-27 21:33:02 on dist-test-slave-pqr8"
server_key: "a35d8ac2cfeab5ddb233e107f6d81e6f"
server_key_iv: "0e6e65f5d2e82d5781b028f0fab986da"
server_key_version: "encryptionkey@0"
I20260927 21:33:02.881265 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e"
format_stamp: "Formatted at 2026-09-27 21:33:02 on dist-test-slave-pqr8"
server_key: "a35d8ac2cfeab5ddb233e107f6d81e6f"
server_key_iv: "0e6e65f5d2e82d5781b028f0fab986da"
server_key_version: "encryptionkey@0"
I20260927 21:33:02.886083 5012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:33:02.889426 5027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:02.890599 5012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 21:33:02.890756 5012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e"
format_stamp: "Formatted at 2026-09-27 21:33:02 on dist-test-slave-pqr8"
server_key: "a35d8ac2cfeab5ddb233e107f6d81e6f"
server_key_iv: "0e6e65f5d2e82d5781b028f0fab986da"
server_key_version: "encryptionkey@0"
I20260927 21:33:02.890944 5012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:02.912456 5012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:02.915964 5012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:02.916213 5012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:02.918817 5012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:02.918919 5012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:02.919071 5012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:02.919176 5012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 21:33:02 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:02.941130 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:02.919209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:47719 (local address 127.3.186.254:43809)
0927 21:33:02.919390 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:02.919396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:02.920772 (+ 1376us) server_negotiation.cc:407] Connection header received
0927 21:33:02.922649 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:02.922656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:02.922770 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:02.922902 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:02.924871 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.925841 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.929078 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.929490 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.929647 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:02.934470 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:02.934479 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:02.934485 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:02.934548 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:02.937765 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:02.938654 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:02.938661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:02.938665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:02.938862 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:02.939460 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:02.939468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:02.939474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:02.939846 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:02.940004 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:02.940611 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 21:33:02.940802 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:02.942286 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:02.919763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.3.186.193:47719)
0927 21:33:02.920570 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:02.920614 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:02.922384 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:02.923177 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:02.923192 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:02.923910 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:02.924707 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:02.924728 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.926034 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.926043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:02.928949 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:02.928962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.929767 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.929779 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:02.930127 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:02.930936 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:02.930965 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:02.934212 (+ 3247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:02.937903 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:02.937914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:02.937936 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:02.938498 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:02.939021 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:02.939027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:02.939033 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:02.939312 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:02.939990 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:02.939996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:02.940290 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 21:33:02.940544 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 21:33:02.940971 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":347,"threads_started":1}
I20260927 21:33:02.961112 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:02.948261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.3.186.193:59251)
0927 21:33:02.948808 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:02.948822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:02.948970 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:02.949286 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:02.949291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:02.949796 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:02.950289 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:02.950306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.951520 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.951525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:02.953171 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:02.953181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.954121 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.954127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:02.954285 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:02.954860 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:02.954875 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:02.955481 (+ 606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:02.958551 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:02.958557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:02.958560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:02.958936 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:02.959269 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:02.959273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:02.959280 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:02.959393 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:02.959946 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:02.959953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:02.960167 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 21:33:02.960323 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 21:33:02.960645 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":320,"threads_started":1}
I20260927 21:33:02.961850 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:02.948196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:59251 (local address 127.3.186.254:43809)
0927 21:33:02.948335 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:02.948339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:02.948904 (+ 565us) server_negotiation.cc:407] Connection header received
0927 21:33:02.949112 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:02.949117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:02.949174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:02.949270 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:02.950445 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.951372 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.953319 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:02.953946 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:02.954083 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:02.955635 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:02.955640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:02.955645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:02.955681 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:02.958432 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:02.959043 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:02.959050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:02.959055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:02.959172 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:02.959501 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:02.959506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:02.959509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:02.959827 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:02.959967 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:02.961374 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0927 21:33:02.961503 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:02.969970 5012 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:40307
I20260927 21:33:02.970006 5146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:40307 every 8 connection(s)
I20260927 21:33:02.971007 5012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:02.972661 5147 heartbeater.cc:347] Connected to a master server at 127.3.186.254:43809
I20260927 21:33:02.972862 5147 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:02.973428 5147 heartbeater.cc:511] Master 127.3.186.254:43809 requested a full tablet report, sending...
I20260927 21:33:02.975162 4941 ts_manager.cc:194] Registered new tserver with Master: 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307)
I20260927 21:33:02.976416 4941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:47719
I20260927 21:33:02.978229 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5012
I20260927 21:33:02.978436 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:02.978992 3819 external_mini_cluster.cc:1428] Setting key 8977a0e8e5c09ff79819cb2ddcf23445
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:33:03.012619 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43809
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:03.306921 5151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:03.307142 5151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:03.307238 5151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:03.313820 5151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:03.313903 5151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:03.313979 5151 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:03.314054 5151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:03.322708 5151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:03.323725 5151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:03.324839 5151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:03.336864 5156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:03.336887 5157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:03.336927 5159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:03.337813 5151 server_base.cc:1034] running on GCE node
I20260927 21:33:03.338274 5151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:03.339003 5151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:03.340231 5151 hybrid_clock.cc:648] HybridClock initialized: now 1790544783340180 us; error 32 us; skew 500 ppm
I20260927 21:33:03.340493 5151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:03 dist-test-slave-pqr8 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:03.343451 5151 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:03.344811 5151 webserver.cc:460] Webserver started at http://127.3.186.194:43369/ using document root <none> and password file <none>
I20260927 21:33:03.345377 5151 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:03.345456 5151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:03.345737 5151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:03.348070 5151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "352f1ac3123140abbbdd157020b0cb33"
format_stamp: "Formatted at 2026-09-27 21:33:03 on dist-test-slave-pqr8"
server_key: "83b8a72696c131d307ad404770c65663"
server_key_iv: "a2d8d01bd4d39d6f28bc33b3ca6bccbb"
server_key_version: "encryptionkey@0"
I20260927 21:33:03.348721 5151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "352f1ac3123140abbbdd157020b0cb33"
format_stamp: "Formatted at 2026-09-27 21:33:03 on dist-test-slave-pqr8"
server_key: "83b8a72696c131d307ad404770c65663"
server_key_iv: "a2d8d01bd4d39d6f28bc33b3ca6bccbb"
server_key_version: "encryptionkey@0"
I20260927 21:33:03.353243 5151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 21:33:03.356238 5166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:03.357604 5151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:33:03.357730 5151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "352f1ac3123140abbbdd157020b0cb33"
format_stamp: "Formatted at 2026-09-27 21:33:03 on dist-test-slave-pqr8"
server_key: "83b8a72696c131d307ad404770c65663"
server_key_iv: "a2d8d01bd4d39d6f28bc33b3ca6bccbb"
server_key_version: "encryptionkey@0"
I20260927 21:33:03.357942 5151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:03.377574 5151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:03.380821 5151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:03.381011 5151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:03.383538 5151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:03.383628 5151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:03.383746 5151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:03.383817 5151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:03 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:03.403134 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.383655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:37255 (local address 127.3.186.254:43809)
0927 21:33:03.383874 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:03.383881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:03.385307 (+ 1426us) server_negotiation.cc:407] Connection header received
0927 21:33:03.387384 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:03.387389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:03.387467 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:03.387592 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:03.389562 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.390297 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.391804 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.392189 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.392317 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.396589 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:03.396598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:03.396605 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:03.396650 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:03.399840 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.400695 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.400703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.400708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.400839 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.401245 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.401251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.401255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.401555 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:03.401713 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:03.402579 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 21:33:03.402807 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:33:03.404011 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.384269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.3.186.194:37255)
0927 21:33:03.385144 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:03.385186 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:03.387061 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:03.387800 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:03.387807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:03.388506 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:03.389365 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.389394 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.390440 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.390445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.391654 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.391665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.392412 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.392420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.392599 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.393348 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:03.393389 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:03.396376 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:03.399990 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.399999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.400012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.400577 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.400946 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.400952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.400955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.401133 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.401690 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:03.401698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:03.402185 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 21:33:03.402507 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 21:33:03.402968 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":349,"threads_started":1}
I20260927 21:33:03.425383 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.409572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:35303 (local address 127.3.186.254:43809)
0927 21:33:03.409761 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:03.409767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:03.410412 (+ 645us) server_negotiation.cc:407] Connection header received
0927 21:33:03.410591 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:03.410597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:03.410662 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:03.410775 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:03.412265 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.412934 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.415477 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.416160 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.416272 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.418782 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:03.418788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:03.418793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:03.418825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:03.422263 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.423199 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.423204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.423208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.423358 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.423792 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.423797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.423801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.424159 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:03.424317 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:03.424843 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 21:33:03.425043 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:33:03.425788 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.409551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.3.186.194:35303)
0927 21:33:03.410315 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:03.410330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:03.410471 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:03.410843 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:03.410851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:03.411551 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:03.412106 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.412120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.413075 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.413082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.415328 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.415345 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.416369 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.416378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.416643 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.417338 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:03.417368 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:03.418585 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:03.422400 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.422407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.422414 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.423060 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.423471 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.423479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.423485 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.423664 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.424291 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:03.424298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:03.424538 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 21:33:03.424725 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 21:33:03.425150 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":453,"threads_started":1}
I20260927 21:33:03.445873 5151 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:33055
I20260927 21:33:03.445907 5285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:33055 every 8 connection(s)
I20260927 21:33:03.447088 5151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:03.448882 5286 heartbeater.cc:347] Connected to a master server at 127.3.186.254:43809
I20260927 21:33:03.449059 5286 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:03.449613 5286 heartbeater.cc:511] Master 127.3.186.254:43809 requested a full tablet report, sending...
I20260927 21:33:03.450688 4941 ts_manager.cc:194] Registered new tserver with Master: 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055)
I20260927 21:33:03.451398 4941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:37255
I20260927 21:33:03.456601 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5151
I20260927 21:33:03.456802 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:03.457197 3819 external_mini_cluster.cc:1428] Setting key a9928d0cbceb1bf92d876a6d5aec7c49
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:33:03.491933 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43809
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:03.790704 5290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:03.790970 5290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:03.791114 5290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:03.798099 5290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:03.798215 5290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:03.798341 5290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:03.798480 5290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:03.807546 5290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:03.808578 5290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:03.809898 5290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:03.821907 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:03.821959 5295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:03.821952 5296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:03.822463 5290 server_base.cc:1034] running on GCE node
I20260927 21:33:03.823038 5290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:03.823828 5290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:03.825071 5290 hybrid_clock.cc:648] HybridClock initialized: now 1790544783825018 us; error 36 us; skew 500 ppm
I20260927 21:33:03.825373 5290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:03 dist-test-slave-pqr8 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:03.833443 5290 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:03.834896 5290 webserver.cc:460] Webserver started at http://127.3.186.195:38855/ using document root <none> and password file <none>
I20260927 21:33:03.835517 5290 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:03.835651 5290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:03.835939 5290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:03.838495 5290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "7ee296fa47ca4953a1ab474c210d1fbf"
format_stamp: "Formatted at 2026-09-27 21:33:03 on dist-test-slave-pqr8"
server_key: "31859e077a1d23dec153e741ed8a3468"
server_key_iv: "dfa7097a7e0e8d04f2f810efac214689"
server_key_version: "encryptionkey@0"
I20260927 21:33:03.839210 5290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "7ee296fa47ca4953a1ab474c210d1fbf"
format_stamp: "Formatted at 2026-09-27 21:33:03 on dist-test-slave-pqr8"
server_key: "31859e077a1d23dec153e741ed8a3468"
server_key_iv: "dfa7097a7e0e8d04f2f810efac214689"
server_key_version: "encryptionkey@0"
I20260927 21:33:03.843598 5290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:33:03.846670 5305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:03.847841 5290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:03.848007 5290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "7ee296fa47ca4953a1ab474c210d1fbf"
format_stamp: "Formatted at 2026-09-27 21:33:03 on dist-test-slave-pqr8"
server_key: "31859e077a1d23dec153e741ed8a3468"
server_key_iv: "dfa7097a7e0e8d04f2f810efac214689"
server_key_version: "encryptionkey@0"
I20260927 21:33:03.848210 5290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:03.861918 5290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:03.867098 5290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:03.867357 5290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:03.870117 5290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:03.870222 5290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:03.870363 5290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:03.870435 5290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:03 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:03.901675 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.872679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:52741 (local address 127.3.186.254:43809)
0927 21:33:03.872847 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:03.872853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:03.873745 (+ 892us) server_negotiation.cc:407] Connection header received
0927 21:33:03.876209 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:03.876215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:03.876302 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:03.876401 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:03.880671 (+ 4270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.881859 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.886254 (+ 4395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.886989 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.887107 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.892202 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:03.892207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:03.892210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:03.892239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:03.896302 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.897345 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.897350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.897354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.897577 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.898388 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.898394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.898399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.898943 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:03.899132 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:03.900702 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0927 21:33:03.901049 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:33:03.904604 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.872094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.3.186.195:52741)
0927 21:33:03.872930 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:03.872968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:03.875876 (+ 2908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:03.878249 (+ 2373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:03.878263 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:03.879375 (+ 1112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:03.880444 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.880473 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.882052 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.882058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.886012 (+ 3954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.886031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.887490 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.887503 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.887828 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.888546 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:03.888570 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:03.891973 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:03.896469 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.896479 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.896496 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.897196 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.897726 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.897731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.897734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.898188 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.899302 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:03.899315 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:03.900092 (+ 777us) client_negotiation.cc:769] Sending connection context
0927 21:33:03.902064 (+ 1972us) client_negotiation.cc:240] Negotiation successful
0927 21:33:03.902845 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":652,"threads_started":1}
I20260927 21:33:03.919661 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:43751 every 8 connection(s)
I20260927 21:33:03.919657 5290 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:43751
I20260927 21:33:03.920959 5290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:03.923131 5425 heartbeater.cc:347] Connected to a master server at 127.3.186.254:43809
I20260927 21:33:03.923621 5425 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:03.924383 5425 heartbeater.cc:511] Master 127.3.186.254:43809 requested a full tablet report, sending...
I20260927 21:33:03.924945 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5290
I20260927 21:33:03.925114 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:03.925695 3819 external_mini_cluster.cc:1428] Setting key 1bafb42d503709f4eb79cd6bc7a01e42
I20260927 21:33:03.926780 4941 ts_manager.cc:194] Registered new tserver with Master: 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751)
I20260927 21:33:03.927816 4941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:52741
I20260927 21:33:03.928706 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:33:03.929589 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.911348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:45241 (local address 127.3.186.254:43809)
0927 21:33:03.911502 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:03.911509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:03.912303 (+ 794us) server_negotiation.cc:407] Connection header received
0927 21:33:03.912546 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:03.912553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:03.912627 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:03.912701 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:03.914191 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.915437 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.917812 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.918537 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.918631 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.921368 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:03.921376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:03.921383 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:03.921424 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:03.925189 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.925938 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.925943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.925946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.926334 (+ 388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.926732 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.926739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.926743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.928374 (+ 1631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:03.928521 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:03.929043 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 21:33:03.929222 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:33:03.929637 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.911452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.3.186.195:45241)
0927 21:33:03.912204 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:03.912226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:03.912404 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:03.912925 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:03.912931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:03.913596 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:03.914013 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.914027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.915645 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.915651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.917612 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:03.917632 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.918967 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.918975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:03.919230 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.919979 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:03.920007 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:03.921156 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:03.925317 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.925322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.925325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.925796 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.926477 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:03.926481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:03.926484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:03.926613 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:03.928499 (+ 1886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:03.928507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:03.928702 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:33:03.928897 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 21:33:03.929188 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":460,"threads_started":1}
Sep 27 21:33:03 dist-test-slave-pqr8 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:33:03 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:03.970748 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.958248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.3.186.254:43809)
0927 21:33:03.958468 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:03.958475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:03.958754 (+ 279us) server_negotiation.cc:407] Connection header received
0927 21:33:03.958888 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:03.958892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:03.958946 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:03.959024 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:03.960199 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.960905 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.962166 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:03.962575 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:03.962665 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:03.966145 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:03.966150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:03.966153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:03.966184 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:03.968351 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.968901 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.968904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.968906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.969015 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:03.969326 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:03.969330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:03.969333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:03.969757 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:03.969974 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:03.970244 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 21:33:03.970440 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:33:03.978794 5147 heartbeater.cc:503] Master 127.3.186.254:43809 was elected leader, sending a full tablet report...
I20260927 21:33:03.974347 4940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:03.980909 4940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:04.016036 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.991685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.3.186.195:43751)
0927 21:33:04.008250 (+ 16565us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.008259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.008292 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:33:04.008394 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.008401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.008647 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.009216 (+ 569us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.009568 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.011030 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.013495 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.014930 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.015051 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.015262 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.015551 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.015698 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16404,"thread_start_us":16255,"threads_started":1}
I20260927 21:33:04.016074 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.991461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:43751 (local address 127.0.0.1:58348)
0927 21:33:04.005877 (+ 14416us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.005917 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.006095 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.008879 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.008886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.008913 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.009390 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.009402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.011154 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.011161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.013299 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.013312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.015114 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.015120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.015310 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.015356 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.015497 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.015659 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14150,"mutex_wait_us":18,"thread_start_us":14705,"threads_started":1}
I20260927 21:33:04.016780 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.991715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:40307 (local address 127.0.0.1:44048)
0927 21:33:04.008268 (+ 16553us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.008288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.008422 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.009112 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.009116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.009141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.009965 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.009977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.011726 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.011731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.014061 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.014074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.016048 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.016057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.016226 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.016254 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.016338 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.016441 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16418,"mutex_wait_us":84,"thread_start_us":16975,"threads_started":1}
I20260927 21:33:04.016885 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.994225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.3.186.194:33055)
0927 21:33:04.001188 (+ 6963us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.001201 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.009107 (+ 7906us) server_negotiation.cc:407] Connection header received
0927 21:33:04.009430 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.009440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.009709 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.009918 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.010766 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.011991 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.014137 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.014947 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.015098 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.015276 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.016481 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.016642 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6788,"thread_start_us":6673,"threads_started":1}
I20260927 21:33:04.017035 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.991882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.3.186.193:40307)
0927 21:33:04.008424 (+ 16542us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.008431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.008452 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:33:04.008694 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.008700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.008908 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.009109 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.010143 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.011588 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.014227 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.015587 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.015715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.015908 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.016616 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.016760 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16415,"thread_start_us":16602,"threads_started":1}
I20260927 21:33:04.017566 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:03.993633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:33055 (local address 127.0.0.1:48544)
0927 21:33:04.008990 (+ 15357us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.009005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.009187 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.010183 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.010187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.010212 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.010550 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.010560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.012116 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.012121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.013939 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.013952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.015410 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.015416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.015587 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.015621 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.017051 (+ 1430us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.017168 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15203,"mutex_wait_us":5,"thread_start_us":14385,"threads_started":1}
I20260927 21:33:04.019985 5077 tablet_service.cc:1467] Processing CreateTablet for tablet cb71438b96364781bd390995c822cdfa (DEFAULT_TABLE table=test-table [id=79cb2e6b40734aedb2c5aeaf320ba2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:04.019944 5355 tablet_service.cc:1467] Processing CreateTablet for tablet cb71438b96364781bd390995c822cdfa (DEFAULT_TABLE table=test-table [id=79cb2e6b40734aedb2c5aeaf320ba2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:04.020704 5221 tablet_service.cc:1467] Processing CreateTablet for tablet cb71438b96364781bd390995c822cdfa (DEFAULT_TABLE table=test-table [id=79cb2e6b40734aedb2c5aeaf320ba2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:04.021109 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb71438b96364781bd390995c822cdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.021293 5355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb71438b96364781bd390995c822cdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.021822 5221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb71438b96364781bd390995c822cdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.031592 5444 tablet_bootstrap.cc:492] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Bootstrap starting.
I20260927 21:33:04.034456 5444 tablet_bootstrap.cc:654] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.036201 5444 log.cc:824] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:04.036512 5445 tablet_bootstrap.cc:492] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Bootstrap starting.
I20260927 21:33:04.037278 5446 tablet_bootstrap.cc:492] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Bootstrap starting.
I20260927 21:33:04.039994 5444 tablet_bootstrap.cc:492] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: No bootstrap required, opened a new log
I20260927 21:33:04.040247 5444 ts_tablet_manager.cc:1397] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 21:33:04.040477 5446 tablet_bootstrap.cc:654] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.040761 5445 tablet_bootstrap.cc:654] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.041365 5446 log.cc:824] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:04.041863 5445 log.cc:824] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:04.043985 5446 tablet_bootstrap.cc:492] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: No bootstrap required, opened a new log
I20260927 21:33:04.044305 5446 ts_tablet_manager.cc:1397] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 21:33:04.045042 5445 tablet_bootstrap.cc:492] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: No bootstrap required, opened a new log
I20260927 21:33:04.045264 5445 ts_tablet_manager.cc:1397] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:33:04.050350 5444 raft_consensus.cc:348] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.050757 5444 raft_consensus.cc:374] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.050894 5444 raft_consensus.cc:729] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d4a99d5e8b4adfaeafaf5e9b39019e, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.051198 5446 raft_consensus.cc:348] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.051549 5446 raft_consensus.cc:374] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.051640 5446 raft_consensus.cc:729] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352f1ac3123140abbbdd157020b0cb33, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.051587 5444 consensus_queue.cc:260] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.052191 5446 consensus_queue.cc:260] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.053332 5444 ts_tablet_manager.cc:1428] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260927 21:33:04.054603 5286 heartbeater.cc:503] Master 127.3.186.254:43809 was elected leader, sending a full tablet report...
I20260927 21:33:04.055220 5446 ts_tablet_manager.cc:1428] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20260927 21:33:04.057919 5445 raft_consensus.cc:348] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.058324 5445 raft_consensus.cc:374] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.058461 5445 raft_consensus.cc:729] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee296fa47ca4953a1ab474c210d1fbf, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.059029 5445 consensus_queue.cc:260] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.060118 5425 heartbeater.cc:503] Master 127.3.186.254:43809 was elected leader, sending a full tablet report...
I20260927 21:33:04.060503 5445 ts_tablet_manager.cc:1428] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 21:33:04.107789 5450 raft_consensus.cc:482] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:04.108033 5450 raft_consensus.cc:504] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.109431 5450 leader_election.cc:288] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751), 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055)
I20260927 21:33:04.116968 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.109694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:45063 (local address 127.3.186.195:43751)
0927 21:33:04.109865 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.109871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.110530 (+ 659us) server_negotiation.cc:407] Connection header received
0927 21:33:04.110697 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.110702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.110764 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.110848 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.111402 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.112098 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.114588 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.116047 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.116205 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.116276 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.116636 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.116743 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:33:04.117142 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.109645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:43751 (local address 127.3.186.193:45063)
0927 21:33:04.110414 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.110437 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.110564 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.110874 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.110878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.110900 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.111254 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.111262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.112267 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.112272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.114427 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.114442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.116253 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.116258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.116454 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.116512 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.116614 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.116727 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":56,"thread_start_us":399,"threads_started":1}
I20260927 21:33:04.117574 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.109864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:33055 (local address 127.3.186.193:40279)
0927 21:33:04.110575 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.110598 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.110735 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.111069 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.111072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.111088 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.111413 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.111422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.113078 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.113084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.115341 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.115355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.116818 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.116824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.116975 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.116993 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.117114 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.117238 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":17,"thread_start_us":405,"threads_started":1}
I20260927 21:33:04.117811 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.109879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:40279 (local address 127.3.186.194:33055)
0927 21:33:04.110092 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.110097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.110717 (+ 620us) server_negotiation.cc:407] Connection header received
0927 21:33:04.110883 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.110887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.110953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.111043 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.111554 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.112933 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.115551 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.116617 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.116790 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.116854 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.117428 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.117580 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:33:04.117996 5375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb71438b96364781bd390995c822cdfa" candidate_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" is_pre_election: true
I20260927 21:33:04.118448 5375 raft_consensus.cc:2454] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d4a99d5e8b4adfaeafaf5e9b39019e in term 0.
I20260927 21:33:04.118525 5241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb71438b96364781bd390995c822cdfa" candidate_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352f1ac3123140abbbdd157020b0cb33" is_pre_election: true
I20260927 21:33:04.118974 5241 raft_consensus.cc:2454] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d4a99d5e8b4adfaeafaf5e9b39019e in term 0.
I20260927 21:33:04.118999 5028 leader_election.cc:302] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d4a99d5e8b4adfaeafaf5e9b39019e, 7ee296fa47ca4953a1ab474c210d1fbf; no voters:
I20260927 21:33:04.119316 5450 raft_consensus.cc:2781] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:04.119436 5450 raft_consensus.cc:482] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:04.119519 5450 raft_consensus.cc:3037] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.121557 5450 raft_consensus.cc:504] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.122115 5450 leader_election.cc:288] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [CANDIDATE]: Term 1 election: Requested vote from peers 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751), 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055)
I20260927 21:33:04.122411 5375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb71438b96364781bd390995c822cdfa" candidate_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee296fa47ca4953a1ab474c210d1fbf"
I20260927 21:33:04.122582 5375 raft_consensus.cc:3037] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.122529 5241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb71438b96364781bd390995c822cdfa" candidate_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352f1ac3123140abbbdd157020b0cb33"
I20260927 21:33:04.122727 5241 raft_consensus.cc:3037] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.124765 5241 raft_consensus.cc:2454] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d4a99d5e8b4adfaeafaf5e9b39019e in term 1.
I20260927 21:33:04.125097 5030 leader_election.cc:302] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d4a99d5e8b4adfaeafaf5e9b39019e, 352f1ac3123140abbbdd157020b0cb33; no voters:
I20260927 21:33:04.125128 5375 raft_consensus.cc:2454] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d4a99d5e8b4adfaeafaf5e9b39019e in term 1.
I20260927 21:33:04.125348 5450 raft_consensus.cc:2781] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:04.125820 5450 raft_consensus.cc:686] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 LEADER]: Becoming Leader. State: Replica: 25d4a99d5e8b4adfaeafaf5e9b39019e, State: Running, Role: LEADER
I20260927 21:33:04.126247 5450 consensus_queue.cc:237] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } }
I20260927 21:33:04.130077 4940 catalog_manager.cc:5495] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e reported cstate change: term changed from 0 to 1, leader changed from <none> to 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193). New cstate: current_term: 1 leader_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:04.170671 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.166106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.3.186.193:40307)
0927 21:33:04.166260 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.166264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.166364 (+ 100us) server_negotiation.cc:407] Connection header received
0927 21:33:04.166574 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.166577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.166646 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.166750 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:04.167003 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.167629 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.168864 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.169237 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.169321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.169778 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.170216 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.170311 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.170453 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.170505 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 21:33:04.172539 5426 tablet.cc:2369] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:04.178287 5241 raft_consensus.cc:1261] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Refusing update from remote peer 25d4a99d5e8b4adfaeafaf5e9b39019e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:04.178297 5375 raft_consensus.cc:1261] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Refusing update from remote peer 25d4a99d5e8b4adfaeafaf5e9b39019e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:04.179162 5455 consensus_queue.cc:1035] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [LEADER]: Connected to new peer: Peer: permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:04.179517 5450 consensus_queue.cc:1035] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:04.193666 5458 mvcc.cc:205] Tried to move back new op lower bound from 7334071435985723392 to 7334071435784355840. Current Snapshot: MvccSnapshot[applied={T|T < 7334071435985723392 or (T in {7334071435985723392})}]
I20260927 21:33:04.207648 4940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:04.207901 4940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:04.210938 4940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: Sending DeleteTablet for 3 replicas of tablet cb71438b96364781bd390995c822cdfa
I20260927 21:33:04.211853 5355 tablet_service.cc:1514] Processing DeleteTablet for tablet cb71438b96364781bd390995c822cdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:04 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:58348
I20260927 21:33:04.212040 5221 tablet_service.cc:1514] Processing DeleteTablet for tablet cb71438b96364781bd390995c822cdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:04 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:48544
I20260927 21:33:04.212479 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet cb71438b96364781bd390995c822cdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:04 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:44048
I20260927 21:33:04.212615 5467 tablet_replica.cc:330] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: stopping tablet replica
I20260927 21:33:04.212960 5467 raft_consensus.cc:2229] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:04.213505 5467 raft_consensus.cc:2258] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:04.215937 5467 ts_tablet_manager.cc:1905] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:04.217132 4940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:04.217777 4940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:04.226576 5076 tablet_service.cc:1467] Processing CreateTablet for tablet 9218a7e7d82b4611846607594f4c5010 (DEFAULT_TABLE table=test-table [id=61ec82c63df044358d14a6382b559eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:04.227300 5076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9218a7e7d82b4611846607594f4c5010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.227928 5469 tablet_replica.cc:330] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: stopping tablet replica
I20260927 21:33:04.228276 5469 raft_consensus.cc:2229] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:04.228843 5469 raft_consensus.cc:2258] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:04.229972 5220 tablet_service.cc:1467] Processing CreateTablet for tablet 9218a7e7d82b4611846607594f4c5010 (DEFAULT_TABLE table=test-table [id=61ec82c63df044358d14a6382b559eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:04.230510 5220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9218a7e7d82b4611846607594f4c5010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.230556 5469 ts_tablet_manager.cc:1905] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:04.234782 5355 tablet_service.cc:1467] Processing CreateTablet for tablet 9218a7e7d82b4611846607594f4c5010 (DEFAULT_TABLE table=test-table [id=61ec82c63df044358d14a6382b559eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:04.235199 5355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9218a7e7d82b4611846607594f4c5010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.238186 5468 tablet_replica.cc:330] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: stopping tablet replica
I20260927 21:33:04.238546 5468 raft_consensus.cc:2229] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:04.238979 5468 raft_consensus.cc:2258] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:04.239593 5469 ts_tablet_manager.cc:1918] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:04.239848 5469 log.cc:1196] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/cb71438b96364781bd390995c822cdfa
I20260927 21:33:04.241061 5468 ts_tablet_manager.cc:1905] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:04.244130 5469 ts_tablet_manager.cc:1939] T cb71438b96364781bd390995c822cdfa P 25d4a99d5e8b4adfaeafaf5e9b39019e: Deleting consensus metadata
I20260927 21:33:04.245970 5444 tablet_bootstrap.cc:492] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Bootstrap starting.
I20260927 21:33:04.246778 4926 catalog_manager.cc:4841] TS 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307): tablet cb71438b96364781bd390995c822cdfa (table test-table [id=79cb2e6b40734aedb2c5aeaf320ba2fb]) successfully deleted
I20260927 21:33:04.247606 5444 tablet_bootstrap.cc:654] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.247890 5446 tablet_bootstrap.cc:492] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Bootstrap starting.
I20260927 21:33:04.248598 5445 tablet_bootstrap.cc:492] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Bootstrap starting.
I20260927 21:33:04.249473 5446 tablet_bootstrap.cc:654] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.250048 5445 tablet_bootstrap.cc:654] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.259615 5444 tablet_bootstrap.cc:492] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: No bootstrap required, opened a new log
I20260927 21:33:04.259845 5444 ts_tablet_manager.cc:1397] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260927 21:33:04.260617 5468 ts_tablet_manager.cc:1918] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:04.260792 5468 log.cc:1196] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/cb71438b96364781bd390995c822cdfa
I20260927 21:33:04.261268 5468 ts_tablet_manager.cc:1939] T cb71438b96364781bd390995c822cdfa P 352f1ac3123140abbbdd157020b0cb33: Deleting consensus metadata
I20260927 21:33:04.261137 5444 raft_consensus.cc:348] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.261534 5444 raft_consensus.cc:374] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.261715 5444 raft_consensus.cc:729] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d4a99d5e8b4adfaeafaf5e9b39019e, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.262110 5444 consensus_queue.cc:260] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.262846 5444 ts_tablet_manager.cc:1428] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:33:04.263419 4927 catalog_manager.cc:4841] TS 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055): tablet cb71438b96364781bd390995c822cdfa (table test-table [id=79cb2e6b40734aedb2c5aeaf320ba2fb]) successfully deleted
I20260927 21:33:04.263447 5467 ts_tablet_manager.cc:1918] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:04.263586 5467 log.cc:1196] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/cb71438b96364781bd390995c822cdfa
I20260927 21:33:04.264112 5467 ts_tablet_manager.cc:1939] T cb71438b96364781bd390995c822cdfa P 7ee296fa47ca4953a1ab474c210d1fbf: Deleting consensus metadata
I20260927 21:33:04.264652 5445 tablet_bootstrap.cc:492] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: No bootstrap required, opened a new log
I20260927 21:33:04.266050 5445 ts_tablet_manager.cc:1397] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20260927 21:33:04.266499 5446 tablet_bootstrap.cc:492] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: No bootstrap required, opened a new log
I20260927 21:33:04.266685 5446 ts_tablet_manager.cc:1397] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.006s
I20260927 21:33:04.266836 5445 raft_consensus.cc:348] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.267088 5445 raft_consensus.cc:374] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.267206 5445 raft_consensus.cc:729] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee296fa47ca4953a1ab474c210d1fbf, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.267484 5445 consensus_queue.cc:260] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.267746 4925 catalog_manager.cc:4841] TS 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751): tablet cb71438b96364781bd390995c822cdfa (table test-table [id=79cb2e6b40734aedb2c5aeaf320ba2fb]) successfully deleted
I20260927 21:33:04.267563 5446 raft_consensus.cc:348] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.267918 5446 raft_consensus.cc:374] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.268062 5446 raft_consensus.cc:729] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352f1ac3123140abbbdd157020b0cb33, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.268075 5445 ts_tablet_manager.cc:1428] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:04.268414 5446 consensus_queue.cc:260] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.269098 5446 ts_tablet_manager.cc:1428] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 21:33:04.386114 5452 raft_consensus.cc:482] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:04.386332 5452 raft_consensus.cc:504] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.387751 5452 leader_election.cc:288] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307), 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055)
I20260927 21:33:04.394903 5451 raft_consensus.cc:482] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:04.395156 5451 raft_consensus.cc:504] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.396646 5451 leader_election.cc:288] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307), 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751)
I20260927 21:33:04.397662 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.388425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:50051 (local address 127.3.186.194:33055)
0927 21:33:04.388587 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.388592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.388829 (+ 237us) server_negotiation.cc:407] Connection header received
0927 21:33:04.389030 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.389037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.389109 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.389189 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.389898 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.390595 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.393324 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.394987 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.395168 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.395253 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.397250 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.397376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:33:04.398697 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.388155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:33055 (local address 127.3.186.195:50051)
0927 21:33:04.388726 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.388741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.388870 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.389352 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.389356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.389379 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.389735 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.389744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.390726 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.390734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.393159 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.393171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.396886 (+ 3715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.396892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.397066 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.397101 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.398194 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.398335 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":160,"thread_start_us":685,"threads_started":1}
I20260927 21:33:04.399083 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.387929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:40307 (local address 127.3.186.195:60965)
0927 21:33:04.388155 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.388175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.388304 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.389087 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.389092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.389136 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.389335 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.389346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.390788 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.390792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.393158 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.393171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.398139 (+ 4968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.398147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.398415 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.398470 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.398572 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.398742 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":220}
I20260927 21:33:04.399636 5241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9218a7e7d82b4611846607594f4c5010" candidate_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352f1ac3123140abbbdd157020b0cb33" is_pre_election: true
I20260927 21:33:04.399849 5241 raft_consensus.cc:2454] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ee296fa47ca4953a1ab474c210d1fbf in term 0.
I20260927 21:33:04.399893 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.388480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:60965 (local address 127.3.186.193:40307)
0927 21:33:04.388657 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.388662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.388680 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:33:04.388756 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.388765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.388846 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.388932 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.389478 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.390661 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.395323 (+ 4662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.396536 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.396673 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.396746 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.399340 (+ 2594us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.399513 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:33:04.400614 5308 leader_election.cc:302] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352f1ac3123140abbbdd157020b0cb33, 7ee296fa47ca4953a1ab474c210d1fbf; no voters:
I20260927 21:33:04.400607 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9218a7e7d82b4611846607594f4c5010" candidate_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" is_pre_election: true
I20260927 21:33:04.400987 5452 raft_consensus.cc:2781] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:04.401077 5452 raft_consensus.cc:482] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:04.401150 5452 raft_consensus.cc:3037] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.401124 5101 raft_consensus.cc:2454] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ee296fa47ca4953a1ab474c210d1fbf in term 0.
I20260927 21:33:04.404593 5452 raft_consensus.cc:504] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.405579 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9218a7e7d82b4611846607594f4c5010" candidate_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e"
I20260927 21:33:04.405807 5101 raft_consensus.cc:3037] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.406231 5241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9218a7e7d82b4611846607594f4c5010" candidate_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352f1ac3123140abbbdd157020b0cb33"
I20260927 21:33:04.406426 5241 raft_consensus.cc:3037] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.409299 5241 raft_consensus.cc:2454] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ee296fa47ca4953a1ab474c210d1fbf in term 1.
I20260927 21:33:04.409682 5101 raft_consensus.cc:2454] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ee296fa47ca4953a1ab474c210d1fbf in term 1.
I20260927 21:33:04.410036 5308 leader_election.cc:302] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352f1ac3123140abbbdd157020b0cb33, 7ee296fa47ca4953a1ab474c210d1fbf; no voters:
I20260927 21:33:04.410574 5452 leader_election.cc:288] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [CANDIDATE]: Term 1 election: Requested vote from peers 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307), 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055)
I20260927 21:33:04.410776 5452 raft_consensus.cc:2781] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:04.411080 5452 raft_consensus.cc:686] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 LEADER]: Becoming Leader. State: Replica: 7ee296fa47ca4953a1ab474c210d1fbf, State: Running, Role: LEADER
I20260927 21:33:04.411613 5452 consensus_queue.cc:237] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.430953 4941 catalog_manager.cc:5495] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195). New cstate: current_term: 1 leader_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } health_report { overall_health: HEALTHY } } }
I20260927 21:33:04.433410 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.397083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:38973 (local address 127.3.186.193:40307)
0927 21:33:04.400571 (+ 3488us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.400577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.419498 (+ 18921us) server_negotiation.cc:407] Connection header received
0927 21:33:04.420632 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.420636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.420695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.420749 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.422274 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.423568 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.431108 (+ 7540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.432234 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.432354 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.432415 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.432945 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.433084 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3362,"thread_start_us":33079,"threads_started":1}
I20260927 21:33:04.434764 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.396741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:40307 (local address 127.3.186.194:38973)
0927 21:33:04.419409 (+ 22668us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.419427 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.420299 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.420934 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.420939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.420966 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.421242 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.421251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.423758 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.423763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.430910 (+ 7147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.430924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.432588 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.432593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.432762 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.432791 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.433957 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.434088 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22556,"thread_start_us":35020,"threads_started":1}
I20260927 21:33:04.437294 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9218a7e7d82b4611846607594f4c5010" candidate_uuid: "352f1ac3123140abbbdd157020b0cb33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" is_pre_election: true
I20260927 21:33:04.437719 5101 raft_consensus.cc:2379] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 352f1ac3123140abbbdd157020b0cb33 in current term 1: Already voted for candidate 7ee296fa47ca4953a1ab474c210d1fbf in this term.
I20260927 21:33:04.437893 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.397896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:48105 (local address 127.3.186.195:43751)
0927 21:33:04.398085 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.398090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.430866 (+ 32776us) server_negotiation.cc:407] Connection header received
0927 21:33:04.431079 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.431084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.431141 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.431190 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:04.431939 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.432850 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.435945 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.436864 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.437032 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.437102 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.437453 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.437620 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 21:33:04.438586 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.397101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:43751 (local address 127.3.186.194:48105)
0927 21:33:04.430782 (+ 33681us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.430799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.430939 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.431340 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.431345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.431365 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:04.431649 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.431659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.433012 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.433016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.435684 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.435699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.437076 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.437083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.437244 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.437286 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.438079 (+ 793us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.438232 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33572,"thread_start_us":28166,"threads_started":1}
I20260927 21:33:04.439426 5375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9218a7e7d82b4611846607594f4c5010" candidate_uuid: "352f1ac3123140abbbdd157020b0cb33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" is_pre_election: true
I20260927 21:33:04.439936 5167 leader_election.cc:302] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 352f1ac3123140abbbdd157020b0cb33; no voters: 25d4a99d5e8b4adfaeafaf5e9b39019e, 7ee296fa47ca4953a1ab474c210d1fbf
I20260927 21:33:04.440279 5451 raft_consensus.cc:2726] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 21:33:04.449793 5287 tablet.cc:2369] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:33:04.473995 5148 tablet.cc:2369] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:33:04 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544782, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:04.488852 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.475490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43809 (local address 127.0.0.1:59454)
0927 21:33:04.476247 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.476264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.476403 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.476714 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.476717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.477363 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:04.477647 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.477656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.478656 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.478663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.479614 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.479625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.480336 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.480342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.480504 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.481140 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:04.481155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:04.483455 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:04.486588 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:04.486600 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:04.486607 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:04.486982 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:04.487305 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:04.487312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:04.487315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:04.487445 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:04.487918 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:04.487925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:04.488070 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.488231 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.488478 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":492,"threads_started":1}
I20260927 21:33:04.488926 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.475501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.3.186.254:43809)
0927 21:33:04.476090 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.476097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.476348 (+ 251us) server_negotiation.cc:407] Connection header received
0927 21:33:04.476524 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.476529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.476588 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.476697 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:04.477792 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.478530 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.479775 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.480157 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.480301 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.483646 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:04.483651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:04.483654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:04.483680 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:04.486452 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.487089 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.487097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.487101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.487207 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.487540 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.487547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.487551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.487786 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:04.487930 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.488304 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.488597 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":427,"thread_start_us":286,"threads_started":1}
I20260927 21:33:04.493296 4941 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:33:04.494437 4941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:04.500887 5076 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf484a525ed4a84a9d9f6b7f3b73461 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0cb71e0f8a8428ca4be82d6b629b7f8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:04.501469 5076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf484a525ed4a84a9d9f6b7f3b73461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.505717 5220 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf484a525ed4a84a9d9f6b7f3b73461 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0cb71e0f8a8428ca4be82d6b629b7f8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:04.506208 5220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf484a525ed4a84a9d9f6b7f3b73461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.506584 5355 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf484a525ed4a84a9d9f6b7f3b73461 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0cb71e0f8a8428ca4be82d6b629b7f8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:04.507134 5355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf484a525ed4a84a9d9f6b7f3b73461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:04.511117 5444 tablet_bootstrap.cc:492] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Bootstrap starting.
I20260927 21:33:04.512807 5446 tablet_bootstrap.cc:492] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: Bootstrap starting.
I20260927 21:33:04.513164 5444 tablet_bootstrap.cc:654] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.514251 5445 tablet_bootstrap.cc:492] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: Bootstrap starting.
I20260927 21:33:04.514475 5446 tablet_bootstrap.cc:654] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.515822 5445 tablet_bootstrap.cc:654] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:04.525466 5446 tablet_bootstrap.cc:492] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: No bootstrap required, opened a new log
I20260927 21:33:04.525547 5444 tablet_bootstrap.cc:492] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: No bootstrap required, opened a new log
I20260927 21:33:04.525689 5446 ts_tablet_manager.cc:1397] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260927 21:33:04.525728 5444 ts_tablet_manager.cc:1397] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260927 21:33:04.526422 5446 raft_consensus.cc:348] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.526436 5444 raft_consensus.cc:348] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.526665 5444 raft_consensus.cc:374] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.526751 5446 raft_consensus.cc:374] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.526788 5444 raft_consensus.cc:729] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d4a99d5e8b4adfaeafaf5e9b39019e, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.526892 5446 raft_consensus.cc:729] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352f1ac3123140abbbdd157020b0cb33, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.527149 5444 consensus_queue.cc:260] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.527212 5446 consensus_queue.cc:260] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.527642 5451 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.527642 5450 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.527901 5451 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.527907 5450 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.528402 5445 tablet_bootstrap.cc:492] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: No bootstrap required, opened a new log
I20260927 21:33:04.528543 5445 ts_tablet_manager.cc:1397] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260927 21:33:04.528797 5446 ts_tablet_manager.cc:1428] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260927 21:33:04.529618 5444 ts_tablet_manager.cc:1428] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:33:04.530243 5445 raft_consensus.cc:348] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.530531 5445 raft_consensus.cc:374] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:04.530627 5445 raft_consensus.cc:729] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee296fa47ca4953a1ab474c210d1fbf, State: Initialized, Role: FOLLOWER
I20260927 21:33:04.530869 5445 consensus_queue.cc:260] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.531384 5445 ts_tablet_manager.cc:1428] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 21:33:04.531266 5452 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.531556 5452 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.562479 5451 raft_consensus.cc:482] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:04.562660 5451 raft_consensus.cc:504] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.563330 5451 leader_election.cc:288] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307), 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751)
I20260927 21:33:04.563771 5375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf484a525ed4a84a9d9f6b7f3b73461" candidate_uuid: "352f1ac3123140abbbdd157020b0cb33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" is_pre_election: true
I20260927 21:33:04.563769 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf484a525ed4a84a9d9f6b7f3b73461" candidate_uuid: "352f1ac3123140abbbdd157020b0cb33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" is_pre_election: true
I20260927 21:33:04.564019 5101 raft_consensus.cc:2454] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352f1ac3123140abbbdd157020b0cb33 in term 0.
I20260927 21:33:04.564029 5375 raft_consensus.cc:2454] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352f1ac3123140abbbdd157020b0cb33 in term 0.
I20260927 21:33:04.564431 5168 leader_election.cc:302] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d4a99d5e8b4adfaeafaf5e9b39019e, 352f1ac3123140abbbdd157020b0cb33; no voters:
I20260927 21:33:04.564721 5451 raft_consensus.cc:2781] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:04.564841 5451 raft_consensus.cc:482] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:04.564947 5451 raft_consensus.cc:3037] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.566920 5451 raft_consensus.cc:504] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.567466 5451 leader_election.cc:288] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [CANDIDATE]: Term 1 election: Requested vote from peers 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307), 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751)
I20260927 21:33:04.567876 5375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf484a525ed4a84a9d9f6b7f3b73461" candidate_uuid: "352f1ac3123140abbbdd157020b0cb33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ee296fa47ca4953a1ab474c210d1fbf"
I20260927 21:33:04.568079 5375 raft_consensus.cc:3037] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.568130 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf484a525ed4a84a9d9f6b7f3b73461" candidate_uuid: "352f1ac3123140abbbdd157020b0cb33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e"
I20260927 21:33:04.568325 5101 raft_consensus.cc:3037] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:04.570958 5375 raft_consensus.cc:2454] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352f1ac3123140abbbdd157020b0cb33 in term 1.
I20260927 21:33:04.571295 5167 leader_election.cc:302] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352f1ac3123140abbbdd157020b0cb33, 7ee296fa47ca4953a1ab474c210d1fbf; no voters:
I20260927 21:33:04.571857 5101 raft_consensus.cc:2454] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352f1ac3123140abbbdd157020b0cb33 in term 1.
I20260927 21:33:04.572124 5451 raft_consensus.cc:2781] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:04.572643 5451 raft_consensus.cc:686] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [term 1 LEADER]: Becoming Leader. State: Replica: 352f1ac3123140abbbdd157020b0cb33, State: Running, Role: LEADER
I20260927 21:33:04.573027 5451 consensus_queue.cc:237] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } }
I20260927 21:33:04.575274 5489 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: TxnStatusTablet state changed. Reason: New leader 352f1ac3123140abbbdd157020b0cb33. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.575630 5489 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:04.576782 5491 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:33:04.577057 5491 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:33:04.577726 4941 catalog_manager.cc:5495] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:04.603055 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.595833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:33055 (local address 127.0.0.1:42044)
0927 21:33:04.596142 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.596162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.596307 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.596804 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.596809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.596821 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:04.597020 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.597030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.598511 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.598516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.600079 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.600094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.601170 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.601176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.601390 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.601475 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.602487 (+ 1012us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.602506 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.602598 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.602742 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 21:33:04.603068 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.595975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.3.186.194:33055)
0927 21:33:04.596128 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.596135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.596228 (+ 93us) server_negotiation.cc:407] Connection header received
0927 21:33:04.596521 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.596527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.596632 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.596777 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:04.597162 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.598345 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.600238 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.600948 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.601113 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.601711 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.602358 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.602503 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.602635 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.602722 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:33:04.607782 5375 raft_consensus.cc:1261] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Refusing update from remote peer 352f1ac3123140abbbdd157020b0cb33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:04.608130 5101 raft_consensus.cc:1261] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Refusing update from remote peer 352f1ac3123140abbbdd157020b0cb33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:04.608354 5451 consensus_queue.cc:1035] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:04.609190 5451 consensus_queue.cc:1035] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:04.622654 5450 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: TxnStatusTablet state changed. Reason: New leader 352f1ac3123140abbbdd157020b0cb33. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.622901 5450 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.625378 5452 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: TxnStatusTablet state changed. Reason: New leader 352f1ac3123140abbbdd157020b0cb33. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.625661 5452 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.626582 5451 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.627252 5451 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:04.627811 5451 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.628062 5451 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 352f1ac3123140abbbdd157020b0cb33: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:04.628576 5459 mvcc.cc:205] Tried to move back new op lower bound from 7334071437746941952 to 7334071437615218688. Current Snapshot: MvccSnapshot[applied={T|T < 7334071437746941952 or (T in {7334071437746941952})}]
I20260927 21:33:04.630529 5460 mvcc.cc:205] Tried to move back new op lower bound from 7334071437746941952 to 7334071437615218688. Current Snapshot: MvccSnapshot[applied={T|T < 7334071437746941952}]
I20260927 21:33:04.631713 5452 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.631949 5452 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 7ee296fa47ca4953a1ab474c210d1fbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.634433 5450 tablet_replica.cc:439] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "352f1ac3123140abbbdd157020b0cb33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 } } peers { permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 } } peers { permanent_uuid: "7ee296fa47ca4953a1ab474c210d1fbf" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 43751 } } }
I20260927 21:33:04.635015 5450 tablet_replica.cc:442] T 1cf484a525ed4a84a9d9f6b7f3b73461 P 25d4a99d5e8b4adfaeafaf5e9b39019e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:04.643236 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.636241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.3.186.195:43751)
0927 21:33:04.636387 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.636395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.636413 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:33:04.636498 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.636503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.636609 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.636741 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:04.637492 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.638564 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.640377 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.640991 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.641115 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.641636 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.642204 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.642349 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.642907 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.642978 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:04.661824 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.654554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:39701 (local address 127.3.186.194:33055)
0927 21:33:04.654703 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.654708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.654910 (+ 202us) server_negotiation.cc:407] Connection header received
0927 21:33:04.655170 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.655175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.655250 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.655362 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:04.655946 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.657487 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.659522 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.660277 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.660383 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.661048 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.661169 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.661226 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.661431 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.661541 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:33:04.662500 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.654177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:33055 (local address 127.3.186.195:39701)
0927 21:33:04.654809 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.654838 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.655032 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.655401 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.655405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.655418 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:04.655744 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.655754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.657691 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.657696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.659343 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.659356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.660573 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.660579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.660793 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.660851 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.661298 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.661307 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.662022 (+ 715us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.662155 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":529,"threads_started":1}
I20260927 21:33:04.673805 5101 raft_consensus.cc:1261] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Refusing update from remote peer 7ee296fa47ca4953a1ab474c210d1fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:04.673905 5241 raft_consensus.cc:1261] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Refusing update from remote peer 7ee296fa47ca4953a1ab474c210d1fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:04.674429 5477 consensus_queue.cc:1035] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "25d4a99d5e8b4adfaeafaf5e9b39019e" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:04.674763 5452 consensus_queue.cc:1035] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "352f1ac3123140abbbdd157020b0cb33" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:04.720893 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.711378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:34025 (local address 127.3.186.195:43751)
0927 21:33:04.711599 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.711607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.715427 (+ 3820us) server_negotiation.cc:407] Connection header received
0927 21:33:04.715695 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.715700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.715772 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.715832 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:04.716680 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.717746 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.719147 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.719611 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.719745 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.720229 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.720350 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.720443 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.720555 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.720623 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:33:04.721487 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.711413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:43751 (local address 127.3.186.194:34025)
0927 21:33:04.715331 (+ 3918us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:04.715353 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:04.715537 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:04.716021 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:04.716025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:04.716037 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:04.716454 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.716468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.717868 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.717873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.719007 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:04.719017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.719817 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.719824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:04.719990 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.720067 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:04.720443 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:04.720462 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 21:33:04.720538 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:33:04.720940 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3728,"thread_start_us":3583,"threads_started":1}
I20260927 21:33:04.754841 4941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:04.755088 4941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:04.758831 4941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfc7a5e3ad94a2f82bb02c68ee3e670: Sending DeleteTablet for 3 replicas of tablet 9218a7e7d82b4611846607594f4c5010
I20260927 21:33:04.759521 5076 tablet_service.cc:1514] Processing DeleteTablet for tablet 9218a7e7d82b4611846607594f4c5010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:04 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:44048
I20260927 21:33:04.759732 5220 tablet_service.cc:1514] Processing DeleteTablet for tablet 9218a7e7d82b4611846607594f4c5010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:04 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:48544
I20260927 21:33:04.760006 5468 tablet_replica.cc:330] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: stopping tablet replica
I20260927 21:33:04.760252 5468 raft_consensus.cc:2229] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:04.760455 5468 raft_consensus.cc:2258] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:04.760540 5355 tablet_service.cc:1514] Processing DeleteTablet for tablet 9218a7e7d82b4611846607594f4c5010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:04 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:58348
I20260927 21:33:04.760805 5467 tablet_replica.cc:330] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: stopping tablet replica
I20260927 21:33:04.761029 5467 raft_consensus.cc:2229] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:04.761489 5467 raft_consensus.cc:2258] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:04.761827 5468 ts_tablet_manager.cc:1905] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:04.766398 5467 ts_tablet_manager.cc:1905] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:04.780566 5467 ts_tablet_manager.cc:1918] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:04.781740 5468 ts_tablet_manager.cc:1918] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:04.782222 5467 log.cc:1196] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/9218a7e7d82b4611846607594f4c5010
I20260927 21:33:04.782969 5467 ts_tablet_manager.cc:1939] T 9218a7e7d82b4611846607594f4c5010 P 7ee296fa47ca4953a1ab474c210d1fbf: Deleting consensus metadata
I20260927 21:33:04.783484 5468 log.cc:1196] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/9218a7e7d82b4611846607594f4c5010
I20260927 21:33:04.784155 5468 ts_tablet_manager.cc:1939] T 9218a7e7d82b4611846607594f4c5010 P 352f1ac3123140abbbdd157020b0cb33: Deleting consensus metadata
I20260927 21:33:04.785336 4925 catalog_manager.cc:4841] TS 7ee296fa47ca4953a1ab474c210d1fbf (127.3.186.195:43751): tablet 9218a7e7d82b4611846607594f4c5010 (table test-table [id=61ec82c63df044358d14a6382b559eb8]) successfully deleted
I20260927 21:33:04.786983 4927 catalog_manager.cc:4841] TS 352f1ac3123140abbbdd157020b0cb33 (127.3.186.194:33055): tablet 9218a7e7d82b4611846607594f4c5010 (table test-table [id=61ec82c63df044358d14a6382b559eb8]) successfully deleted
I20260927 21:33:04.792986 5506 tablet_replica.cc:330] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: stopping tablet replica
I20260927 21:33:04.793310 5506 raft_consensus.cc:2229] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:04.793646 5506 raft_consensus.cc:2258] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:04.795181 5506 ts_tablet_manager.cc:1905] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:04.802289 5506 ts_tablet_manager.cc:1918] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:04.802464 5506 log.cc:1196] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/9218a7e7d82b4611846607594f4c5010
I20260927 21:33:04.802973 5506 ts_tablet_manager.cc:1939] T 9218a7e7d82b4611846607594f4c5010 P 25d4a99d5e8b4adfaeafaf5e9b39019e: Deleting consensus metadata
I20260927 21:33:04.803987 4926 catalog_manager.cc:4841] TS 25d4a99d5e8b4adfaeafaf5e9b39019e (127.3.186.193:40307): tablet 9218a7e7d82b4611846607594f4c5010 (table test-table [id=61ec82c63df044358d14a6382b559eb8]) successfully deleted
Sep 27 21:33:04 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:04.817912 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.771925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.3.186.193:40307)
0927 21:33:04.772094 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.772100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.794441 (+ 22341us) server_negotiation.cc:407] Connection header received
0927 21:33:04.799131 (+ 4690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.799137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.799218 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.799293 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:04.801814 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.802951 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.804866 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.805501 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.805667 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.810621 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:04.810636 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:04.810655 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:04.810696 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:04.814527 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.815325 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.815335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.815339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.815494 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.815889 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.815894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.815897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.816300 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:04.816600 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.816880 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.817575 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 21:33:04.818605 5142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50724
I20260927 21:33:04.837833 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.822265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.3.186.254:43809)
0927 21:33:04.822453 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.822462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.822819 (+ 357us) server_negotiation.cc:407] Connection header received
0927 21:33:04.822970 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.822976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.823060 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.823159 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:04.824805 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.825928 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.827662 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.828675 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.828799 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.831358 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:04.831365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:04.831370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:04.831409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:04.834762 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.835465 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.835471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.835474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.835616 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.835980 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.835984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.835988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.836298 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:04.836530 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.837263 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.837473 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 21:33:04.838587 4941 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59456
I20260927 21:33:04.840498 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5012
I20260927 21:33:04.857313 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.841588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.3.186.193:40307)
0927 21:33:04.841756 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.841762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.842965 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 21:33:04.843118 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.843123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.843201 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.843301 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:04.844757 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.845917 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.847737 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.848328 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.848450 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.851075 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:04.851082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:04.851087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:04.851123 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:04.854602 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.855323 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.855328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.855332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.855467 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.855855 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.855860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.855864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.856194 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:04.856456 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.856704 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.856921 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 21:33:04.857937 5142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734
W20260927 21:33:04.858474 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:04.858636 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5012
I20260927 21:33:04.886991 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5151
Sep 27 21:33:04 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:04.906210 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.888262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.3.186.194:33055)
0927 21:33:04.888425 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.888430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.888458 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:33:04.888513 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.888518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.888592 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.888699 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:04.890167 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.891445 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.893227 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.893971 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.894102 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.898976 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:04.898987 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:04.899003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:04.899047 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:04.903021 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.903777 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.903786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.903789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.903967 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.904358 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.904363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.904367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.904775 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:04.905085 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.905383 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.905644 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 21:33:04.906921 5281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42046
W20260927 21:33:04.907648 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5151: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:04.907809 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5151
I20260927 21:33:04.925910 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5290
Sep 27 21:33:04 dist-test-slave-pqr8 krb5kdc[4896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:04.944643 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:04.926704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.3.186.195:43751)
0927 21:33:04.926873 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:04.926879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:04.926896 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:33:04.926948 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:04.926953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:04.927033 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:04.927136 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:04.928525 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.929838 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.931509 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:04.932153 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:04.932282 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:04.937409 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:04.937422 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:04.937438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:04.937483 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:04.941353 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.942105 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.942114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.942118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.942255 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:04.942638 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:04.942643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:04.942647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:04.943080 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:04.943375 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:04.944054 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 21:33:04.944247 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 21:33:04.945300 5420 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49014
W20260927 21:33:04.946007 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5290: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:04.946141 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5290
I20260927 21:33:04.963781 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 4909
I20260927 21:33:04.964668 4981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:05.073585 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 4909
2026-09-27T21:33:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4254 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:05 dist-test-slave-pqr8 krb5kdc[5515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:05 dist-test-slave-pqr8 krb5kdc[5515](info): set up 2 sockets
Sep 27 21:33:05 dist-test-slave-pqr8 krb5kdc[5515](info): commencing operation
krb5kdc: starting...
W20260927 21:33:06.117354 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:06 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:06.279094 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41211
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:06.570997 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:06.571257 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:06.571386 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:06.578025 5528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:06.578102 5528 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:33:06.578173 5528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:06.578277 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:06.578354 5528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:06.578464 5528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:06.587919 5528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:41211
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:06.588967 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:06.590180 5528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:06.601037 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:06.601194 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:06.601745 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:06.601989 5528 server_base.cc:1034] running on GCE node
I20260927 21:33:06.602708 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:06.603917 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:06.605121 5528 hybrid_clock.cc:648] HybridClock initialized: now 1790544786605072 us; error 32 us; skew 500 ppm
I20260927 21:33:06.605417 5528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:06 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:06.608232 5528 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:06.609503 5528 webserver.cc:460] Webserver started at http://127.3.186.254:32913/ using document root <none> and password file <none>
I20260927 21:33:06.610158 5528 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:06.610286 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:06.610569 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:06.613186 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "f7f4ed691c144f62b3be6299b9a2ae8d"
format_stamp: "Formatted at 2026-09-27 21:33:06 on dist-test-slave-pqr8"
server_key: "8c7fc71b71a020c8d9d2f0b408b6af5c"
server_key_iv: "44f86029da6cd84a245792cbec407664"
server_key_version: "encryptionkey@0"
I20260927 21:33:06.613924 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "f7f4ed691c144f62b3be6299b9a2ae8d"
format_stamp: "Formatted at 2026-09-27 21:33:06 on dist-test-slave-pqr8"
server_key: "8c7fc71b71a020c8d9d2f0b408b6af5c"
server_key_iv: "44f86029da6cd84a245792cbec407664"
server_key_version: "encryptionkey@0"
I20260927 21:33:06.618345 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:06.621255 5543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:06.622380 5528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:06.622540 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "f7f4ed691c144f62b3be6299b9a2ae8d"
format_stamp: "Formatted at 2026-09-27 21:33:06 on dist-test-slave-pqr8"
server_key: "8c7fc71b71a020c8d9d2f0b408b6af5c"
server_key_iv: "44f86029da6cd84a245792cbec407664"
server_key_version: "encryptionkey@0"
I20260927 21:33:06.622735 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:06.649631 5528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:06.653062 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:06.653301 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:06.673004 5528 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:41211
I20260927 21:33:06.673024 5604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:41211 every 8 connection(s)
I20260927 21:33:06.674201 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:06.677390 5605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:06.682068 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5528
I20260927 21:33:06.682250 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:06.683019 3819 external_mini_cluster.cc:1428] Setting key a655ed315b8a0ae2f3f8da9e229c8576
I20260927 21:33:06.688035 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Bootstrap starting.
I20260927 21:33:06.690987 5605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:06.691898 5605 log.cc:824] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:06.694518 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: No bootstrap required, opened a new log
Sep 27 21:33:06 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:06.702415 5605 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41211 } }
I20260927 21:33:06.702832 5605 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:06.702955 5605 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f4ed691c144f62b3be6299b9a2ae8d, State: Initialized, Role: FOLLOWER
I20260927 21:33:06.703539 5605 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41211 } }
I20260927 21:33:06.703801 5605 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:06.703815 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:06.685006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.3.186.254:41211)
0927 21:33:06.685859 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:06.685869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:06.685909 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:33:06.687470 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:06.687488 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:06.687813 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:06.688159 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:06.689564 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:06.690853 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:06.692134 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:06.692942 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:06.693128 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:06.696882 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:06.696893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:06.696919 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:06.696970 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:06.699961 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:06.700669 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:06.700678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:06.700683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:06.700850 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:06.701212 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:06.701218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:06.701221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:06.701752 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:06.702018 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:06.702381 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 21:33:06.702725 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":369,"threads_started":1}
I20260927 21:33:06.703934 5605 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:06.704082 5605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:06.706696 5605 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41211 } }
I20260927 21:33:06.707096 5605 leader_election.cc:302] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f4ed691c144f62b3be6299b9a2ae8d; no voters:
I20260927 21:33:06.707536 5605 leader_election.cc:288] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:06.707628 5610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:06.708359 5610 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [term 1 LEADER]: Becoming Leader. State: Replica: f7f4ed691c144f62b3be6299b9a2ae8d, State: Running, Role: LEADER
I20260927 21:33:06.708848 5610 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41211 } }
I20260927 21:33:06.709400 5605 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:06.711951 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41211 } } }
I20260927 21:33:06.712332 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:06.711964 5612 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f4ed691c144f62b3be6299b9a2ae8d. Latest consensus state: current_term: 1 leader_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4ed691c144f62b3be6299b9a2ae8d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41211 } } }
I20260927 21:33:06.712503 5612 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:06.718034 5618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:06.722585 5618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:06.730298 5618 catalog_manager.cc:1347] Generated new cluster ID: 7230404eb8294c23a7e293bdb876f9b7
I20260927 21:33:06.730386 5618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:06.754487 5618 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:06.756042 5618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:06.764532 5618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Generated new TSK 0
I20260927 21:33:06.765364 5618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:06.807147 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41211
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:07.107834 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:07.108057 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:07.108176 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:07.114987 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:07.115077 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:07.115204 5631 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:33:07.115262 5631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:07.115399 5631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:07.124589 5631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:41211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:07.125722 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:07.126998 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:07.139683 5636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:07.139819 5637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:07.139847 5639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:07.140462 5631 server_base.cc:1034] running on GCE node
I20260927 21:33:07.141036 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:07.141880 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:07.143083 5631 hybrid_clock.cc:648] HybridClock initialized: now 1790544787143040 us; error 52 us; skew 500 ppm
I20260927 21:33:07.143388 5631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:07 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:07.146236 5631 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:07.147485 5631 webserver.cc:460] Webserver started at http://127.3.186.193:40557/ using document root <none> and password file <none>
I20260927 21:33:07.148104 5631 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:07.148237 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:07.148515 5631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:07.151165 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "f4269b1b258842f9a2c47b69dcb4ff6a"
format_stamp: "Formatted at 2026-09-27 21:33:07 on dist-test-slave-pqr8"
server_key: "c24c1ff09e28bdcdcf92a55a63b25b55"
server_key_iv: "533705aefc7c883895b7c9373a74f409"
server_key_version: "encryptionkey@0"
I20260927 21:33:07.151911 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "f4269b1b258842f9a2c47b69dcb4ff6a"
format_stamp: "Formatted at 2026-09-27 21:33:07 on dist-test-slave-pqr8"
server_key: "c24c1ff09e28bdcdcf92a55a63b25b55"
server_key_iv: "533705aefc7c883895b7c9373a74f409"
server_key_version: "encryptionkey@0"
I20260927 21:33:07.156515 5631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 21:33:07.159471 5646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:07.160583 5631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:33:07.160737 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "f4269b1b258842f9a2c47b69dcb4ff6a"
format_stamp: "Formatted at 2026-09-27 21:33:07 on dist-test-slave-pqr8"
server_key: "c24c1ff09e28bdcdcf92a55a63b25b55"
server_key_iv: "533705aefc7c883895b7c9373a74f409"
server_key_version: "encryptionkey@0"
I20260927 21:33:07.160919 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:07.203730 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:07.207253 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:07.207516 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:07.209911 5631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:07.210052 5631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:07.210201 5631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:07.210301 5631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:07 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:07.232993 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.210497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:47879 (local address 127.3.186.254:41211)
0927 21:33:07.213042 (+ 2545us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:07.213047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:07.213064 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:33:07.215367 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:07.215372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:07.215458 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:07.215570 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:07.217635 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.218997 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.221556 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.222221 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.222375 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.226847 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:07.226851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:07.226854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:07.226884 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:07.229393 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.230419 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.230426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.230431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.230590 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.231138 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.231146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.231150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.231449 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:07.231599 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:07.232430 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 21:33:07.232639 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2443,"thread_start_us":2350,"threads_started":1}
I20260927 21:33:07.234063 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.211224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.3.186.193:47879)
0927 21:33:07.212221 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:07.212269 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:07.214983 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:07.215852 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:07.215865 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:07.216623 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:07.217351 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.217367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.219165 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.219169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.221380 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.221400 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.222574 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.222581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.222738 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.223341 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:07.223361 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:07.226614 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:07.229545 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.229556 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.229572 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.230271 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.230733 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.230737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.230741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.230945 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.231581 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:07.231588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:07.231996 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 21:33:07.232335 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 21:33:07.232800 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":466,"threads_started":1}
I20260927 21:33:07.254703 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.239663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.3.186.193:34735)
0927 21:33:07.240274 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:07.240290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:07.240479 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:07.240962 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:07.240966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:07.241367 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:07.241848 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.241865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.243090 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.243093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.244716 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.244730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.245857 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.245863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.246015 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.246726 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:07.246740 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:07.247815 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:07.251951 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.251959 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.251966 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.252395 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.252753 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.252756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.252758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.252865 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.253480 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:07.253489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:07.253741 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 21:33:07.253917 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 21:33:07.254135 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":386,"threads_started":1}
I20260927 21:33:07.255352 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.239560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:34735 (local address 127.3.186.254:41211)
0927 21:33:07.239687 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:07.239690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:07.240366 (+ 676us) server_negotiation.cc:407] Connection header received
0927 21:33:07.240612 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:07.240616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:07.240706 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:07.240799 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:07.242004 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.242891 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.244870 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.245444 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.245621 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.248030 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:07.248039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:07.248046 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:07.248102 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:07.251796 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.252503 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.252508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.252511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.252646 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.252963 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.252968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.252971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.253356 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:07.253538 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:07.255015 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0927 21:33:07.255158 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:33:07.263131 5631 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:44205
I20260927 21:33:07.263230 5766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:44205 every 8 connection(s)
I20260927 21:33:07.264438 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:07.266256 5767 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41211
I20260927 21:33:07.266420 5767 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:07.266995 5767 heartbeater.cc:511] Master 127.3.186.254:41211 requested a full tablet report, sending...
I20260927 21:33:07.268580 5560 ts_manager.cc:194] Registered new tserver with Master: f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205)
I20260927 21:33:07.269997 5560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:47879
I20260927 21:33:07.271147 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5631
I20260927 21:33:07.271327 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:07.271924 3819 external_mini_cluster.cc:1428] Setting key e86635dab40297e7e5b88f704998717f
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:33:07.306896 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41211
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:07.602103 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:07.602437 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:07.602625 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:07.609622 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:07.609738 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:07.609887 5771 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:33:07.609959 5771 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:07.610054 5771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:07.618896 5771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:41211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:07.619908 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:07.621074 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:07.633464 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:07.633543 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:07.633533 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:07.634163 5771 server_base.cc:1034] running on GCE node
I20260927 21:33:07.634708 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:07.635465 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:07.636693 5771 hybrid_clock.cc:648] HybridClock initialized: now 1790544787636641 us; error 26 us; skew 500 ppm
I20260927 21:33:07.637034 5771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:07 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:07.639818 5771 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:07.641011 5771 webserver.cc:460] Webserver started at http://127.3.186.194:41725/ using document root <none> and password file <none>
I20260927 21:33:07.641611 5771 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:07.641764 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:07.642072 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:07.644640 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "38c642981a404976afcf528da52cb681"
format_stamp: "Formatted at 2026-09-27 21:33:07 on dist-test-slave-pqr8"
server_key: "c859da7b0774d293f558212fd6f5bd93"
server_key_iv: "52748586833d1c32909bc3d9c8092821"
server_key_version: "encryptionkey@0"
I20260927 21:33:07.645330 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "38c642981a404976afcf528da52cb681"
format_stamp: "Formatted at 2026-09-27 21:33:07 on dist-test-slave-pqr8"
server_key: "c859da7b0774d293f558212fd6f5bd93"
server_key_iv: "52748586833d1c32909bc3d9c8092821"
server_key_version: "encryptionkey@0"
I20260927 21:33:07.649755 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:07.652801 5786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:07.654023 5771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:07.654174 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "38c642981a404976afcf528da52cb681"
format_stamp: "Formatted at 2026-09-27 21:33:07 on dist-test-slave-pqr8"
server_key: "c859da7b0774d293f558212fd6f5bd93"
server_key_iv: "52748586833d1c32909bc3d9c8092821"
server_key_version: "encryptionkey@0"
I20260927 21:33:07.654354 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:07.679756 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:07.683966 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:07.684242 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:07.686743 5771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:07.686892 5771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:33:07.687034 5771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:07.687182 5771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:07 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:07.710131 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.687282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:60823 (local address 127.3.186.254:41211)
0927 21:33:07.687442 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:07.687449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:07.689019 (+ 1570us) server_negotiation.cc:407] Connection header received
0927 21:33:07.691732 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:07.691741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:07.691859 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:07.691991 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:07.693902 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.694941 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.697165 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.697790 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.697929 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.703020 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:07.703026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:07.703029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:07.703058 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:07.706756 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.707609 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.707617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.707623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.707823 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.708314 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.708319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.708323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.708696 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:07.708876 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:07.709584 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 21:33:07.709780 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:33:07.711138 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.687987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.3.186.194:60823)
0927 21:33:07.688880 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:07.688926 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:07.691415 (+ 2489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:07.692201 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:07.692211 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:07.692849 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:07.693684 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.693712 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.695114 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.695123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.697016 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.697029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.698057 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.698068 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.698434 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.699265 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:07.699294 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:07.702714 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:07.706902 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.706912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.706930 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.707474 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.707937 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.707943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.707949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.708188 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.708854 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:07.708861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:07.709193 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 21:33:07.709455 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 21:33:07.709943 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":401,"threads_started":1}
I20260927 21:33:07.734858 5771 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:40717
I20260927 21:33:07.734879 5905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:40717 every 8 connection(s)
I20260927 21:33:07.736459 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:07.738698 5906 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41211
I20260927 21:33:07.738914 5906 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:07.739614 5906 heartbeater.cc:511] Master 127.3.186.254:41211 requested a full tablet report, sending...
I20260927 21:33:07.739956 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.724135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:56345 (local address 127.3.186.254:41211)
0927 21:33:07.724336 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:07.724344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:07.725101 (+ 757us) server_negotiation.cc:407] Connection header received
0927 21:33:07.725305 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:07.725312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:07.725395 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:07.725495 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:07.726935 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.728091 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.730550 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.731285 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.731434 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.733786 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:07.733792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:07.733796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:07.733828 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:07.737218 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.738121 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.738128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.738131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.738266 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:07.738649 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:07.738654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:07.738660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:07.739021 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:07.739134 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:07.739542 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 21:33:07.739712 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:33:07.740217 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:07.724328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.3.186.194:56345)
0927 21:33:07.724976 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:07.724999 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:07.725140 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:07.725575 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:07.725581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:07.726394 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:07.726756 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.726787 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.728253 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.728258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.730393 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:07.730409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:07.731491 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:07.731508 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:07.731718 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:07.732589 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:07.732615 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:07.733569 (+ 954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:07.737386 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.737391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.737395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.737939 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.738370 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:07.738374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:07.738378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:07.738530 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:07.739120 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:07.739124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:07.739318 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 21:33:07.739473 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 21:33:07.739766 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":339,"threads_started":1}
I20260927 21:33:07.740927 5560 ts_manager.cc:194] Registered new tserver with Master: 38c642981a404976afcf528da52cb681 (127.3.186.194:40717)
I20260927 21:33:07.741393 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5771
I20260927 21:33:07.741576 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:07.741984 5560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:60823
I20260927 21:33:07.742080 3819 external_mini_cluster.cc:1428] Setting key e273f0512d5ef8b9df720b05fcdf97b9
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:33:07.778570 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41211
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:08.075249 5910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:08.075497 5910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:08.075618 5910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:08.082199 5910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:08.082309 5910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:08.082458 5910 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:33:08.082551 5910 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:08.082660 5910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:08.091346 5910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:41211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:08.092370 5910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:08.093636 5910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:08.105559 5918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:08.105585 5916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:08.106109 5915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:08.106582 5910 server_base.cc:1034] running on GCE node
I20260927 21:33:08.107105 5910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:08.107913 5910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:08.109119 5910 hybrid_clock.cc:648] HybridClock initialized: now 1790544788109075 us; error 37 us; skew 500 ppm
I20260927 21:33:08.109411 5910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:08 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:08.112540 5910 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:08.113968 5910 webserver.cc:460] Webserver started at http://127.3.186.195:45643/ using document root <none> and password file <none>
I20260927 21:33:08.114542 5910 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:08.114669 5910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:08.114933 5910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:08.117321 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "97fed1090f2a40c9a88509ea574fa893"
format_stamp: "Formatted at 2026-09-27 21:33:08 on dist-test-slave-pqr8"
server_key: "490bcdbb5437c6b6498699ce00517456"
server_key_iv: "3f636d7b846703783a6cdc34fc14d79d"
server_key_version: "encryptionkey@0"
I20260927 21:33:08.118012 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "97fed1090f2a40c9a88509ea574fa893"
format_stamp: "Formatted at 2026-09-27 21:33:08 on dist-test-slave-pqr8"
server_key: "490bcdbb5437c6b6498699ce00517456"
server_key_iv: "3f636d7b846703783a6cdc34fc14d79d"
server_key_version: "encryptionkey@0"
I20260927 21:33:08.122444 5910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:33:08.125447 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:08.126765 5910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:08.126883 5910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "97fed1090f2a40c9a88509ea574fa893"
format_stamp: "Formatted at 2026-09-27 21:33:08 on dist-test-slave-pqr8"
server_key: "490bcdbb5437c6b6498699ce00517456"
server_key_iv: "3f636d7b846703783a6cdc34fc14d79d"
server_key_version: "encryptionkey@0"
I20260927 21:33:08.127031 5910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:08.139986 5910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:08.143786 5910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:08.143999 5910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:08.146349 5910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:08.146492 5910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:08.146610 5910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:08.146693 5910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:08 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:08.171007 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.147108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.3.186.195:59771)
0927 21:33:08.147883 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.147944 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.150552 (+ 2608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.151291 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.151299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.152019 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:08.152782 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.152801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.154549 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.154556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.156669 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.156679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.157994 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.158006 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.158329 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.159051 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:08.159078 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:08.161879 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:08.166327 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:08.166337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:08.166360 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:08.166934 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:08.167412 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:08.167418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:08.167422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:08.167694 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:08.168348 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:08.168361 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:08.168788 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.169088 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.169581 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":312,"threads_started":1}
I20260927 21:33:08.171970 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.146630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:59771 (local address 127.3.186.254:41211)
0927 21:33:08.146800 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.146806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.148070 (+ 1264us) server_negotiation.cc:407] Connection header received
0927 21:33:08.150839 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.150844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.150906 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.151047 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:08.153005 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.154363 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.156786 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.157670 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.157798 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.162111 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:08.162118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:08.162124 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:08.162166 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:08.166155 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.167065 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.167072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.167078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.167239 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.167806 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.167814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.167818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.168202 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:08.168343 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.171497 (+ 3154us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.171738 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:33:08.194921 5910 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:42613
I20260927 21:33:08.195007 6044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:42613 every 8 connection(s)
I20260927 21:33:08.195600 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.180536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:34105 (local address 127.3.186.254:41211)
0927 21:33:08.180668 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.180671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.181481 (+ 810us) server_negotiation.cc:407] Connection header received
0927 21:33:08.181693 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.181715 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.181775 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.181882 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:08.183284 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.184178 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.186588 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.187081 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.187223 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.189802 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:08.189810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:08.189815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:08.189865 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:08.192761 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.193711 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.193719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.193726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.193884 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.194275 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.194278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.194280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.194483 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:08.194643 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.195071 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.195264 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:08.195693 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.180856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.3.186.195:34105)
0927 21:33:08.181388 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.181404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.181566 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.182047 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.182051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.182678 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:08.183097 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.183116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.184354 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.184364 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.186421 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.186438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.187334 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.187343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.187586 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.188272 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:08.188295 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:08.189611 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:08.192895 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:08.192903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:08.192909 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:08.193567 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:08.194017 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:08.194022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:08.194025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:08.194162 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:08.194603 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:08.194608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:08.194785 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.194970 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.195277 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":314,"threads_started":1}
I20260927 21:33:08.196450 5910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:08.199180 6045 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41211
I20260927 21:33:08.199419 6045 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:08.199975 6045 heartbeater.cc:511] Master 127.3.186.254:41211 requested a full tablet report, sending...
I20260927 21:33:08.201073 5560 ts_manager.cc:194] Registered new tserver with Master: 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:08.201813 5560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:59771
I20260927 21:33:08.205329 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 5910
I20260927 21:33:08.205451 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:08.205863 3819 external_mini_cluster.cc:1428] Setting key 6321e7917e1dec9c63acb3e42a7b5e7c
I20260927 21:33:08.208221 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:33:08 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:33:08 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:08.250017 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.238149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.3.186.254:41211)
0927 21:33:08.238345 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.238353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.238499 (+ 146us) server_negotiation.cc:407] Connection header received
0927 21:33:08.238632 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.238636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.238688 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.238761 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:08.239902 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.240576 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.241753 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.242126 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.242227 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.245691 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:08.245695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:08.245698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:08.245725 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:08.247839 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.248325 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.248329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.248331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.248428 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.248742 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.248745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.248747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.248971 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:08.249132 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.249439 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.249658 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:33:08.253696 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:08.260311 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:08.272424 5767 heartbeater.cc:503] Master 127.3.186.254:41211 was elected leader, sending a full tablet report...
I20260927 21:33:08.283071 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.272215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.3.186.194:40717)
0927 21:33:08.274070 (+ 1855us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.274081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.274680 (+ 599us) server_negotiation.cc:407] Connection header received
0927 21:33:08.275622 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.275629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.275865 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.276020 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.276970 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.278327 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.280416 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.281760 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.281897 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.282074 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.282566 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.282735 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1664,"thread_start_us":1504,"threads_started":1}
I20260927 21:33:08.283156 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.271717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.3.186.195:42613)
0927 21:33:08.273676 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.273687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.274702 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 21:33:08.275097 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.275108 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.275374 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.275554 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.276314 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.277822 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.280468 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.281777 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.281896 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.282116 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.282657 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.282815 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":1507,"threads_started":1}
I20260927 21:33:08.283655 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.275738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:44205 (local address 127.0.0.1:47190)
0927 21:33:08.276551 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.276564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.276711 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.277262 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.277266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.277288 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.277607 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.277617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.279285 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.279290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.281056 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.281065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.282756 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.282761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.282918 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.282948 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.283061 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.283222 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":5,"thread_start_us":548,"threads_started":1}
I20260927 21:33:08.283731 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.271586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:40717 (local address 127.0.0.1:52970)
0927 21:33:08.274560 (+ 2974us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.274592 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.275432 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.276264 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.276271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.276298 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.276804 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.276819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.278492 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.278496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.280257 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.280268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.281945 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.281951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.282312 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.282376 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.283232 (+ 856us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.283347 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2846,"mutex_wait_us":46,"thread_start_us":2521,"threads_started":1}
I20260927 21:33:08.284192 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.275008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.3.186.193:44205)
0927 21:33:08.276122 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.276130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.276660 (+ 530us) server_negotiation.cc:407] Connection header received
0927 21:33:08.276864 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.276873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.277083 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.277259 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.277764 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.279143 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.281201 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.282303 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.282428 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.282625 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.283782 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.283936 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":864,"threads_started":1}
I20260927 21:33:08.284845 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.271455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:42613 (local address 127.0.0.1:44026)
0927 21:33:08.274572 (+ 3117us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.274622 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.274839 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.275647 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.275653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.275679 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.276091 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.276104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.277932 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.277937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.280277 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.280293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.282252 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.282258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.282449 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.282479 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.283376 (+ 897us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.283503 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2812,"mutex_wait_us":66,"thread_start_us":2836,"threads_started":1}
I20260927 21:33:08.286979 5836 tablet_service.cc:1467] Processing CreateTablet for tablet c9ae62de6a8d4c65bc3a02ae34fb5002 (DEFAULT_TABLE table=test-table [id=7dd5d0143ad549088e99fa2e7f8587ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:08.287153 5697 tablet_service.cc:1467] Processing CreateTablet for tablet c9ae62de6a8d4c65bc3a02ae34fb5002 (DEFAULT_TABLE table=test-table [id=7dd5d0143ad549088e99fa2e7f8587ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:08.288220 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ae62de6a8d4c65bc3a02ae34fb5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:08.288344 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ae62de6a8d4c65bc3a02ae34fb5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:08.288055 5975 tablet_service.cc:1467] Processing CreateTablet for tablet c9ae62de6a8d4c65bc3a02ae34fb5002 (DEFAULT_TABLE table=test-table [id=7dd5d0143ad549088e99fa2e7f8587ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:08.289227 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ae62de6a8d4c65bc3a02ae34fb5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:08.297896 6064 tablet_bootstrap.cc:492] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Bootstrap starting.
I20260927 21:33:08.301462 6065 tablet_bootstrap.cc:492] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Bootstrap starting.
I20260927 21:33:08.301595 6064 tablet_bootstrap.cc:654] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:08.302634 6066 tablet_bootstrap.cc:492] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Bootstrap starting.
I20260927 21:33:08.302804 6064 log.cc:824] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:08.305277 6065 tablet_bootstrap.cc:654] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:08.305424 6066 tablet_bootstrap.cc:654] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:08.305783 6064 tablet_bootstrap.cc:492] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: No bootstrap required, opened a new log
I20260927 21:33:08.306020 6064 ts_tablet_manager.cc:1397] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:33:08.306403 6066 log.cc:824] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:08.306466 6065 log.cc:824] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:08.308771 6066 tablet_bootstrap.cc:492] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: No bootstrap required, opened a new log
I20260927 21:33:08.309072 6066 ts_tablet_manager.cc:1397] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 21:33:08.315469 6065 tablet_bootstrap.cc:492] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: No bootstrap required, opened a new log
I20260927 21:33:08.315697 6065 ts_tablet_manager.cc:1397] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260927 21:33:08.316319 6066 raft_consensus.cc:348] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.316615 6066 raft_consensus.cc:374] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:08.316720 6066 raft_consensus.cc:729] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fed1090f2a40c9a88509ea574fa893, State: Initialized, Role: FOLLOWER
I20260927 21:33:08.317220 6066 consensus_queue.cc:260] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.317427 6064 raft_consensus.cc:348] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.317896 6064 raft_consensus.cc:374] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:08.318020 6064 raft_consensus.cc:729] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c642981a404976afcf528da52cb681, State: Initialized, Role: FOLLOWER
I20260927 21:33:08.318159 6045 heartbeater.cc:503] Master 127.3.186.254:41211 was elected leader, sending a full tablet report...
I20260927 21:33:08.318737 6066 ts_tablet_manager.cc:1428] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260927 21:33:08.318619 6064 consensus_queue.cc:260] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.319527 5906 heartbeater.cc:503] Master 127.3.186.254:41211 was elected leader, sending a full tablet report...
I20260927 21:33:08.320127 6064 ts_tablet_manager.cc:1428] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 21:33:08.326337 6065 raft_consensus.cc:348] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.326632 6065 raft_consensus.cc:374] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:08.326766 6065 raft_consensus.cc:729] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Initialized, Role: FOLLOWER
I20260927 21:33:08.327193 6065 consensus_queue.cc:260] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.328508 6065 ts_tablet_manager.cc:1428] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260927 21:33:08.440330 6071 raft_consensus.cc:482] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:08.440594 6071 raft_consensus.cc:504] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.442154 6071 leader_election.cc:288] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613), f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205)
I20260927 21:33:08.448359 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.442434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:33163 (local address 127.3.186.195:42613)
0927 21:33:08.442610 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.442614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.443149 (+ 535us) server_negotiation.cc:407] Connection header received
0927 21:33:08.443328 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.443336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.443414 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.443514 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.444331 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.445042 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.446757 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.447475 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.447662 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.447731 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.447997 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.448111 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:33:08.448433 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.442368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:42613 (local address 127.3.186.194:33163)
0927 21:33:08.442997 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.443041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.443173 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.443718 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.443725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.443773 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.444181 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.444192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.445167 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.445174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.446605 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.446616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.447692 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.447696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.447843 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.447864 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.447961 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.448084 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":313,"threads_started":1}
W20260927 21:33:08.448738 6046 tablet.cc:2369] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:08.448807 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.443424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:44545 (local address 127.3.186.193:44205)
0927 21:33:08.443609 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.443616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.444051 (+ 435us) server_negotiation.cc:407] Connection header received
0927 21:33:08.444219 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.444222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.444286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.444390 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.444926 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.445669 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.447266 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.447985 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.448153 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.448232 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.448488 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.448598 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:33:08.448853 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.443167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:44205 (local address 127.3.186.194:44545)
0927 21:33:08.443943 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.443965 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.444088 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.444403 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.444407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.444431 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.444756 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.444764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.445794 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.445801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.447114 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.447125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.448184 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.448202 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.448336 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.448356 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.448459 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.448566 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":451,"threads_started":1}
I20260927 21:33:08.449859 5722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ae62de6a8d4c65bc3a02ae34fb5002" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" is_pre_election: true
I20260927 21:33:08.449908 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ae62de6a8d4c65bc3a02ae34fb5002" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893" is_pre_election: true
I20260927 21:33:08.450309 5722 raft_consensus.cc:2454] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 0.
I20260927 21:33:08.450341 5995 raft_consensus.cc:2454] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 0.
I20260927 21:33:08.450922 5787 leader_election.cc:302] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:08.451272 6071 raft_consensus.cc:2781] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:08.451416 6071 raft_consensus.cc:482] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:08.451527 6071 raft_consensus.cc:3037] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:08.453646 6071 raft_consensus.cc:504] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.454180 6071 leader_election.cc:288] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 election: Requested vote from peers 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613), f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205)
I20260927 21:33:08.454604 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ae62de6a8d4c65bc3a02ae34fb5002" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893"
I20260927 21:33:08.454667 5722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ae62de6a8d4c65bc3a02ae34fb5002" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a"
I20260927 21:33:08.454784 5995 raft_consensus.cc:3037] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:08.454865 5722 raft_consensus.cc:3037] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:08.456808 5995 raft_consensus.cc:2454] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 1.
I20260927 21:33:08.456897 5722 raft_consensus.cc:2454] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 1.
I20260927 21:33:08.457188 5787 leader_election.cc:302] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, 97fed1090f2a40c9a88509ea574fa893; no voters:
I20260927 21:33:08.457505 6071 raft_consensus.cc:2781] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:08.458034 6071 raft_consensus.cc:686] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 1 LEADER]: Becoming Leader. State: Replica: 38c642981a404976afcf528da52cb681, State: Running, Role: LEADER
I20260927 21:33:08.458527 6071 consensus_queue.cc:237] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } }
I20260927 21:33:08.462541 5560 catalog_manager.cc:5495] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c642981a404976afcf528da52cb681 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "38c642981a404976afcf528da52cb681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } health_report { overall_health: UNKNOWN } } }
W20260927 21:33:08.488466 5907 tablet.cc:2369] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:33:08.516613 5768 tablet.cc:2369] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:33:08 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:08.535171 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.523379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.3.186.254:41211)
0927 21:33:08.523526 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.523530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.524064 (+ 534us) server_negotiation.cc:407] Connection header received
0927 21:33:08.524241 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.524246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.524321 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.524412 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:08.525430 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.526142 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.527224 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.527608 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.527728 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.530561 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:08.530567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:08.530571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:08.530601 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:08.532686 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.533305 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.533313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.533316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.533442 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:08.533826 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:08.533831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:08.533834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:08.534084 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:08.534222 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.534590 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.534880 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":50}
I20260927 21:33:08.535203 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.523463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41211 (local address 127.0.0.1:60552)
0927 21:33:08.523954 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.523968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.524129 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.524430 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.524434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.525054 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:08.525300 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.525309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.526248 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.526255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.527103 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.527114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.527760 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.527766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.527907 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.528451 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:08.528466 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:08.530397 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:08.532821 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:08.532831 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:08.532837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:08.533200 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:08.533560 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:08.533564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:08.533567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:08.533691 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:08.534214 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:08.534221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:08.534355 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.534509 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.534764 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260927 21:33:08.539538 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:60552:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:33:08.540634 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:08.546284 5697 tablet_service.cc:1467] Processing CreateTablet for tablet c5376171520f44939f07f4e48379d855 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bdcf56ef8e04437a1f6371da5d15b29]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:08.546590 5975 tablet_service.cc:1467] Processing CreateTablet for tablet c5376171520f44939f07f4e48379d855 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bdcf56ef8e04437a1f6371da5d15b29]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:08.546655 5836 tablet_service.cc:1467] Processing CreateTablet for tablet c5376171520f44939f07f4e48379d855 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bdcf56ef8e04437a1f6371da5d15b29]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:08.547183 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5376171520f44939f07f4e48379d855. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:08.547341 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5376171520f44939f07f4e48379d855. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:08.547381 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5376171520f44939f07f4e48379d855. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:08.552111 6066 tablet_bootstrap.cc:492] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: Bootstrap starting.
I20260927 21:33:08.553501 6066 tablet_bootstrap.cc:654] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:08.553963 6065 tablet_bootstrap.cc:492] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: Bootstrap starting.
I20260927 21:33:08.554072 6064 tablet_bootstrap.cc:492] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: Bootstrap starting.
I20260927 21:33:08.555434 6065 tablet_bootstrap.cc:654] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:08.555694 6064 tablet_bootstrap.cc:654] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:08.555788 6066 tablet_bootstrap.cc:492] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: No bootstrap required, opened a new log
I20260927 21:33:08.555990 6066 ts_tablet_manager.cc:1397] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 21:33:08.556903 6066 raft_consensus.cc:348] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.557241 6066 raft_consensus.cc:374] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:08.557379 6066 raft_consensus.cc:729] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fed1090f2a40c9a88509ea574fa893, State: Initialized, Role: FOLLOWER
I20260927 21:33:08.557709 6066 consensus_queue.cc:260] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.558028 6065 tablet_bootstrap.cc:492] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: No bootstrap required, opened a new log
I20260927 21:33:08.558149 6065 ts_tablet_manager.cc:1397] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:33:08.558168 6064 tablet_bootstrap.cc:492] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: No bootstrap required, opened a new log
I20260927 21:33:08.558254 6066 ts_tablet_manager.cc:1428] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:33:08.558267 6064 ts_tablet_manager.cc:1397] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:33:08.558115 6070 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.558437 6070 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.558807 6065 raft_consensus.cc:348] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.558883 6064 raft_consensus.cc:348] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.559011 6065 raft_consensus.cc:374] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:08.559108 6065 raft_consensus.cc:729] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Initialized, Role: FOLLOWER
I20260927 21:33:08.559098 6064 raft_consensus.cc:374] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:08.559218 6064 raft_consensus.cc:729] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c642981a404976afcf528da52cb681, State: Initialized, Role: FOLLOWER
I20260927 21:33:08.559314 6065 consensus_queue.cc:260] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.559382 6064 consensus_queue.cc:260] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.559839 6065 ts_tablet_manager.cc:1428] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:08.559876 6064 ts_tablet_manager.cc:1428] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 21:33:08.560642 6071 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.560845 6071 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.561056 6072 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.561268 6072 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.724287 6072 raft_consensus.cc:482] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:08.724483 6072 raft_consensus.cc:504] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.725849 6072 leader_election.cc:288] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38c642981a404976afcf528da52cb681 (127.3.186.194:40717), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:08.732718 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.726269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:47191 (local address 127.3.186.195:42613)
0927 21:33:08.726413 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.726419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.727138 (+ 719us) server_negotiation.cc:407] Connection header received
0927 21:33:08.727432 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.727438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.727513 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.727610 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.728383 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.729165 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.731119 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.731841 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.732009 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.732077 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.732361 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.732470 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:33:08.732837 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.726236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:38067 (local address 127.3.186.194:40717)
0927 21:33:08.726431 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.726438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.727360 (+ 922us) server_negotiation.cc:407] Connection header received
0927 21:33:08.727456 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.727461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.727526 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.727623 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:08.728358 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.729079 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.731181 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.731944 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.732129 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.732200 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.732469 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.732585 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 21:33:08.732873 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.726332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:42613 (local address 127.3.186.193:47191)
0927 21:33:08.726965 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.726988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.727276 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.727643 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.727648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.727674 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.728169 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.728186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.729292 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.729296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.731009 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.731020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.732031 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.732037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.732212 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.732234 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.732329 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.732447 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":84,"thread_start_us":313,"threads_started":1}
I20260927 21:33:08.732986 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.726341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:40717 (local address 127.3.186.193:38067)
0927 21:33:08.727245 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.727260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.727363 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.727643 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.727648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.727730 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:08.728207 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.728218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.729221 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.729229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.731035 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.731050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.732163 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.732172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.732325 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.732345 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.732445 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.732560 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":130,"spinlock_wait_cycles":21376,"thread_start_us":523,"threads_started":1}
I20260927 21:33:08.733500 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5376171520f44939f07f4e48379d855" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893" is_pre_election: true
I20260927 21:33:08.733780 5995 raft_consensus.cc:2454] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 0.
I20260927 21:33:08.733889 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5376171520f44939f07f4e48379d855" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c642981a404976afcf528da52cb681" is_pre_election: true
I20260927 21:33:08.734237 5647 leader_election.cc:302] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97fed1090f2a40c9a88509ea574fa893, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:08.734329 5860 raft_consensus.cc:2454] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 0.
I20260927 21:33:08.734618 6072 raft_consensus.cc:2781] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:08.734736 6072 raft_consensus.cc:482] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:08.734817 6072 raft_consensus.cc:3037] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:08.737834 6072 raft_consensus.cc:504] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.738757 6072 leader_election.cc:288] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 election: Requested vote from peers 38c642981a404976afcf528da52cb681 (127.3.186.194:40717), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:08.738854 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5376171520f44939f07f4e48379d855" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c642981a404976afcf528da52cb681"
I20260927 21:33:08.738833 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5376171520f44939f07f4e48379d855" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893"
I20260927 21:33:08.739033 5995 raft_consensus.cc:3037] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:08.739028 5860 raft_consensus.cc:3037] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:08.741808 5995 raft_consensus.cc:2454] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 1.
I20260927 21:33:08.741991 5860 raft_consensus.cc:2454] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 1.
I20260927 21:33:08.742158 5647 leader_election.cc:302] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97fed1090f2a40c9a88509ea574fa893, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:08.742421 6072 raft_consensus.cc:2781] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:08.742842 6072 raft_consensus.cc:686] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 LEADER]: Becoming Leader. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Running, Role: LEADER
I20260927 21:33:08.743345 6072 consensus_queue.cc:237] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:08.746744 6088 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: TxnStatusTablet state changed. Reason: New leader f4269b1b258842f9a2c47b69dcb4ff6a. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.747143 6088 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:08.747790 6090 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:33:08.748062 6090 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:33:08.748430 5559 catalog_manager.cc:5495] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193). New cstate: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:08.796875 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.791058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:44205 (local address 127.0.0.1:47194)
0927 21:33:08.791267 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.791283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.791387 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.791878 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.791882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.791891 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:08.792030 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.792037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.793447 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.793451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.794472 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.794483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.795408 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.795412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.795563 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.795610 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.796520 (+ 910us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.796531 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.796578 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.796674 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:33:08.797009 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.791142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.3.186.193:44205)
0927 21:33:08.791302 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.791308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.791327 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:33:08.791584 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.791590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.791707 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.791850 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:08.792154 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.793314 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.794610 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.795269 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.795404 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.795786 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.796422 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.796566 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.796611 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.796691 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:33:08.803706 5995 raft_consensus.cc:1261] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Refusing update from remote peer f4269b1b258842f9a2c47b69dcb4ff6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:08.803731 5860 raft_consensus.cc:1261] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Refusing update from remote peer f4269b1b258842f9a2c47b69dcb4ff6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:08.804589 6088 consensus_queue.cc:1035] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:08.804951 6072 consensus_queue.cc:1035] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:08.819137 6071 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: TxnStatusTablet state changed. Reason: New leader f4269b1b258842f9a2c47b69dcb4ff6a. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.819491 6071 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.822121 6093 mvcc.cc:205] Tried to move back new op lower bound from 7334071454931644416 to 7334071454696562688. Current Snapshot: MvccSnapshot[applied={T|T < 7334071454931644416}]
I20260927 21:33:08.826002 6070 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: TxnStatusTablet state changed. Reason: New leader f4269b1b258842f9a2c47b69dcb4ff6a. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.826217 6070 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.830832 6088 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.831064 6088 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:08.837031 6071 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.837283 6071 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P 38c642981a404976afcf528da52cb681: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.838581 6095 mvcc.cc:205] Tried to move back new op lower bound from 7334071454931644416 to 7334071454696562688. Current Snapshot: MvccSnapshot[applied={T|T < 7334071454931644416 or (T in {7334071454931644416})}]
I20260927 21:33:08.841650 6072 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.841907 6072 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P f4269b1b258842f9a2c47b69dcb4ff6a: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:08.844264 6094 mvcc.cc:205] Tried to move back new op lower bound from 7334071454931644416 to 7334071454696562688. Current Snapshot: MvccSnapshot[applied={T|T < 7334071454931644416}]
I20260927 21:33:08.856702 6071 consensus_queue.cc:1035] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:08.860442 6070 tablet_replica.cc:439] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } } }
I20260927 21:33:08.860729 6070 tablet_replica.cc:442] T c5376171520f44939f07f4e48379d855 P 97fed1090f2a40c9a88509ea574fa893: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:08.868232 6071 consensus_queue.cc:1035] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:08.883198 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.874997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.3.186.194:40717)
0927 21:33:08.875203 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.875212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.878148 (+ 2936us) server_negotiation.cc:407] Connection header received
0927 21:33:08.878383 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.878387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.878452 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.878596 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:08.879026 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.879816 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.881129 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.881586 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.881708 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.882133 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.882601 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.882753 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.882808 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.882881 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:33:08.898100 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.892627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:60431 (local address 127.3.186.193:44205)
0927 21:33:08.892785 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.892792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.893407 (+ 615us) server_negotiation.cc:407] Connection header received
0927 21:33:08.893626 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.893631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.893719 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.893829 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:08.894324 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.895029 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.896460 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.896883 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.897017 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.897415 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.897542 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.897623 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.897759 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.897821 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:08.898216 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.892656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:44205 (local address 127.3.186.194:60431)
0927 21:33:08.893300 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.893327 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.893475 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.893842 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.893847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.893857 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:08.894164 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.894171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.895167 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.895173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.896303 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.896314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.897053 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.897059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.897188 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.897231 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.897624 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.897636 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.897740 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.897851 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":353,"threads_started":1}
I20260927 21:33:08.947901 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.939312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:35127 (local address 127.3.186.194:40717)
0927 21:33:08.939455 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:08.939460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:08.940253 (+ 793us) server_negotiation.cc:407] Connection header received
0927 21:33:08.940407 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:08.940414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:08.940515 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:08.940613 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:08.941276 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.942389 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.945322 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.946178 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.946315 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.947031 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.947172 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.947234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:08.947453 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:33:08.947539 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:33:08.948787 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:08.939309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:40717 (local address 127.3.186.193:35127)
0927 21:33:08.940141 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:08.940160 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:08.940284 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:08.940638 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:08.940643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:08.940655 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:08.941051 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.941062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.942861 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.942866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.945163 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:08.945178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:08.946486 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:08.946493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:08.946736 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:08.946812 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:08.947317 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:08.947330 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:33:08.948090 (+ 760us) client_negotiation.cc:240] Negotiation successful
0927 21:33:08.948279 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":503,"threads_started":1}
I20260927 21:33:08.998306 5560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:08.998564 5560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:09.002326 5560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Sending DeleteTablet for 3 replicas of tablet c9ae62de6a8d4c65bc3a02ae34fb5002
I20260927 21:33:09.003917 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ae62de6a8d4c65bc3a02ae34fb5002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:08 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260927 21:33:09.004549 5836 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ae62de6a8d4c65bc3a02ae34fb5002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:08 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:52970
I20260927 21:33:09.005576 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ae62de6a8d4c65bc3a02ae34fb5002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:08 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:44026
I20260927 21:33:09.012668 6115 tablet_replica.cc:330] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: stopping tablet replica
I20260927 21:33:09.013178 6115 raft_consensus.cc:2229] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:09.013818 6116 tablet_replica.cc:330] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: stopping tablet replica
I20260927 21:33:09.013880 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:33:09.014358 6116 raft_consensus.cc:2229] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:09.014693 6115 raft_consensus.cc:2258] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 21:33:09.014739 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:09.015872 6116 raft_consensus.cc:2258] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:09.020385 6115 ts_tablet_manager.cc:1905] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:09.023070 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 71328582fd1a4a29bfb142dd416c62d2 (DEFAULT_TABLE table=test-table [id=6f6dec2956d94f2c925fee6aba90da21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:09.023550 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71328582fd1a4a29bfb142dd416c62d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:09.026805 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 71328582fd1a4a29bfb142dd416c62d2 (DEFAULT_TABLE table=test-table [id=6f6dec2956d94f2c925fee6aba90da21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:09.027247 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71328582fd1a4a29bfb142dd416c62d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:09.027297 6115 ts_tablet_manager.cc:1918] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:09.027413 6115 log.cc:1196] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/c9ae62de6a8d4c65bc3a02ae34fb5002
I20260927 21:33:09.027966 6115 ts_tablet_manager.cc:1939] T c9ae62de6a8d4c65bc3a02ae34fb5002 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting consensus metadata
I20260927 21:33:09.029842 5544 catalog_manager.cc:4841] TS f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205): tablet c9ae62de6a8d4c65bc3a02ae34fb5002 (table test-table [id=7dd5d0143ad549088e99fa2e7f8587ef]) successfully deleted
I20260927 21:33:09.031049 6116 ts_tablet_manager.cc:1905] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:09.034324 5836 tablet_service.cc:1467] Processing CreateTablet for tablet 71328582fd1a4a29bfb142dd416c62d2 (DEFAULT_TABLE table=test-table [id=6f6dec2956d94f2c925fee6aba90da21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:09.035382 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71328582fd1a4a29bfb142dd416c62d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:09.039462 6066 tablet_bootstrap.cc:492] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Bootstrap starting.
I20260927 21:33:09.040371 6065 tablet_bootstrap.cc:492] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Bootstrap starting.
I20260927 21:33:09.041055 6066 tablet_bootstrap.cc:654] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:09.042083 6065 tablet_bootstrap.cc:654] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:09.045347 6117 tablet_replica.cc:330] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: stopping tablet replica
I20260927 21:33:09.045751 6117 raft_consensus.cc:2229] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:09.046142 6117 raft_consensus.cc:2258] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:09.048818 6117 ts_tablet_manager.cc:1905] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:09.053642 6065 tablet_bootstrap.cc:492] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: No bootstrap required, opened a new log
I20260927 21:33:09.053851 6065 ts_tablet_manager.cc:1397] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260927 21:33:09.054600 6065 raft_consensus.cc:348] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.056718 6117 ts_tablet_manager.cc:1918] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:09.056896 6117 log.cc:1196] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/c9ae62de6a8d4c65bc3a02ae34fb5002
I20260927 21:33:09.057338 6065 raft_consensus.cc:374] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:09.057475 6065 raft_consensus.cc:729] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Initialized, Role: FOLLOWER
I20260927 21:33:09.058223 6117 ts_tablet_manager.cc:1939] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 97fed1090f2a40c9a88509ea574fa893: Deleting consensus metadata
I20260927 21:33:09.058426 6064 tablet_bootstrap.cc:492] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Bootstrap starting.
I20260927 21:33:09.058386 6065 consensus_queue.cc:260] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.060037 6064 tablet_bootstrap.cc:654] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:09.060457 6116 ts_tablet_manager.cc:1918] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:09.060595 6116 log.cc:1196] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/c9ae62de6a8d4c65bc3a02ae34fb5002
I20260927 21:33:09.061136 6116 ts_tablet_manager.cc:1939] T c9ae62de6a8d4c65bc3a02ae34fb5002 P 38c642981a404976afcf528da52cb681: Deleting consensus metadata
I20260927 21:33:09.061761 6065 ts_tablet_manager.cc:1428] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260927 21:33:09.063324 5544 catalog_manager.cc:4841] TS 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613): tablet c9ae62de6a8d4c65bc3a02ae34fb5002 (table test-table [id=7dd5d0143ad549088e99fa2e7f8587ef]) successfully deleted
I20260927 21:33:09.064193 6066 tablet_bootstrap.cc:492] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: No bootstrap required, opened a new log
I20260927 21:33:09.064204 5545 catalog_manager.cc:4841] TS 38c642981a404976afcf528da52cb681 (127.3.186.194:40717): tablet c9ae62de6a8d4c65bc3a02ae34fb5002 (table test-table [id=7dd5d0143ad549088e99fa2e7f8587ef]) successfully deleted
I20260927 21:33:09.064402 6066 ts_tablet_manager.cc:1397] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.008s
I20260927 21:33:09.065306 6066 raft_consensus.cc:348] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.065563 6066 raft_consensus.cc:374] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:09.065645 6066 raft_consensus.cc:729] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fed1090f2a40c9a88509ea574fa893, State: Initialized, Role: FOLLOWER
I20260927 21:33:09.073657 6066 consensus_queue.cc:260] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.074196 6064 tablet_bootstrap.cc:492] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: No bootstrap required, opened a new log
I20260927 21:33:09.074311 6066 ts_tablet_manager.cc:1428] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20260927 21:33:09.074446 6064 ts_tablet_manager.cc:1397] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260927 21:33:09.075387 6064 raft_consensus.cc:348] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.075714 6064 raft_consensus.cc:374] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:09.075845 6064 raft_consensus.cc:729] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c642981a404976afcf528da52cb681, State: Initialized, Role: FOLLOWER
I20260927 21:33:09.076187 6064 consensus_queue.cc:260] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.079279 6064 ts_tablet_manager.cc:1428] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 21:33:09.205906 6088 raft_consensus.cc:482] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:09.206142 6088 raft_consensus.cc:504] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.207257 6088 leader_election.cc:288] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38c642981a404976afcf528da52cb681 (127.3.186.194:40717), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:09.208084 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71328582fd1a4a29bfb142dd416c62d2" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893" is_pre_election: true
I20260927 21:33:09.208353 5994 raft_consensus.cc:2454] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 0.
I20260927 21:33:09.210093 5647 leader_election.cc:302] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97fed1090f2a40c9a88509ea574fa893, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:09.209378 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71328582fd1a4a29bfb142dd416c62d2" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c642981a404976afcf528da52cb681" is_pre_election: true
I20260927 21:33:09.210569 5860 raft_consensus.cc:2454] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 0.
I20260927 21:33:09.211386 6088 raft_consensus.cc:2781] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:09.211493 6088 raft_consensus.cc:482] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:09.211606 6088 raft_consensus.cc:3037] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:09.215582 6088 raft_consensus.cc:504] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.216749 5860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71328582fd1a4a29bfb142dd416c62d2" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c642981a404976afcf528da52cb681"
I20260927 21:33:09.216972 5860 raft_consensus.cc:3037] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:09.220537 5860 raft_consensus.cc:2454] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 1.
I20260927 21:33:09.221132 5648 leader_election.cc:302] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:09.221483 6072 raft_consensus.cc:2781] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:09.221643 6072 raft_consensus.cc:686] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 LEADER]: Becoming Leader. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Running, Role: LEADER
I20260927 21:33:09.221947 6072 consensus_queue.cc:237] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:09.227406 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71328582fd1a4a29bfb142dd416c62d2" candidate_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893"
I20260927 21:33:09.227622 5994 raft_consensus.cc:3037] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:09.231683 5994 raft_consensus.cc:2454] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4269b1b258842f9a2c47b69dcb4ff6a in term 1.
I20260927 21:33:09.232453 6088 leader_election.cc:288] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [CANDIDATE]: Term 1 election: Requested vote from peers 38c642981a404976afcf528da52cb681 (127.3.186.194:40717), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:09.247285 5560 catalog_manager.cc:5495] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193). New cstate: current_term: 1 leader_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:09.294714 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:09.288468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.3.186.193:44205)
0927 21:33:09.288670 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:09.288678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:09.288697 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:33:09.288762 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:09.288768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:09.288877 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:09.288966 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:09.289338 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:09.290483 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:09.292337 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:09.293011 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:09.293138 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:09.293692 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:09.293831 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:09.293905 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:09.294385 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 21:33:09.294452 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:33:09.313619 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:09.307494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:44205 (local address 127.3.186.193:41103)
0927 21:33:09.307755 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:09.307772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:09.307880 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:09.308334 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:09.308338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:09.308348 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:09.308508 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:09.308517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:09.309982 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:09.309987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:09.311606 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:09.311619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:09.312501 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:09.312506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:09.312704 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:09.312744 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:09.313120 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:09.313126 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:33:09.313189 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:33:09.313307 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 21:33:09.314267 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:09.307457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:41103 (local address 127.3.186.193:44205)
0927 21:33:09.307618 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:09.307636 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:09.308006 (+ 370us) server_negotiation.cc:407] Connection header received
0927 21:33:09.308046 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:09.308060 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:09.308133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:09.308192 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:09.308647 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:09.309780 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:09.311742 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:09.312319 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:09.312415 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:09.312892 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:09.313011 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:09.313085 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:09.313958 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 21:33:09.314023 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:09.327319 5860 raft_consensus.cc:1261] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Refusing update from remote peer f4269b1b258842f9a2c47b69dcb4ff6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:09.327869 6072 consensus_queue.cc:1035] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:09.329828 5994 raft_consensus.cc:1261] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Refusing update from remote peer f4269b1b258842f9a2c47b69dcb4ff6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:09.330817 6072 consensus_queue.cc:1035] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:33:09.362573 6093 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:33:10.944939 5560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:10.945168 5560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:10.948553 5560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Sending DeleteTablet for 3 replicas of tablet 71328582fd1a4a29bfb142dd416c62d2
I20260927 21:33:10.949117 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 71328582fd1a4a29bfb142dd416c62d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:10 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260927 21:33:10.949141 5836 tablet_service.cc:1514] Processing DeleteTablet for tablet 71328582fd1a4a29bfb142dd416c62d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:10 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:52970
I20260927 21:33:10.949273 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 71328582fd1a4a29bfb142dd416c62d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:10 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:44026
I20260927 21:33:10.960031 6145 tablet_replica.cc:330] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: stopping tablet replica
I20260927 21:33:10.960351 6145 raft_consensus.cc:2229] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:10.960623 6145 raft_consensus.cc:2258] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:10.961161 6144 tablet_replica.cc:330] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: stopping tablet replica
I20260927 21:33:10.961344 6144 raft_consensus.cc:2229] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:10.961755 6146 tablet_replica.cc:330] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: stopping tablet replica
I20260927 21:33:10.961990 6146 raft_consensus.cc:2229] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:10.962031 6145 ts_tablet_manager.cc:1905] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:10.962224 6146 raft_consensus.cc:2258] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:10.962857 6144 raft_consensus.cc:2258] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:10.963395 6146 ts_tablet_manager.cc:1905] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:10.963810 6144 ts_tablet_manager.cc:1905] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:10.968345 6145 ts_tablet_manager.cc:1918] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:33:10.968513 6145 log.cc:1196] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/71328582fd1a4a29bfb142dd416c62d2
I20260927 21:33:10.968968 6145 ts_tablet_manager.cc:1939] T 71328582fd1a4a29bfb142dd416c62d2 P 38c642981a404976afcf528da52cb681: Deleting consensus metadata
I20260927 21:33:10.969583 6146 ts_tablet_manager.cc:1918] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:33:10.969749 6146 log.cc:1196] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/71328582fd1a4a29bfb142dd416c62d2
I20260927 21:33:10.969995 5545 catalog_manager.cc:4841] TS 38c642981a404976afcf528da52cb681 (127.3.186.194:40717): tablet 71328582fd1a4a29bfb142dd416c62d2 (table test-table [id=6f6dec2956d94f2c925fee6aba90da21]) successfully deleted
I20260927 21:33:10.970228 6146 ts_tablet_manager.cc:1939] T 71328582fd1a4a29bfb142dd416c62d2 P 97fed1090f2a40c9a88509ea574fa893: Deleting consensus metadata
I20260927 21:33:10.970773 6144 ts_tablet_manager.cc:1918] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:33:10.970929 6144 log.cc:1196] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/71328582fd1a4a29bfb142dd416c62d2
I20260927 21:33:10.970996 5544 catalog_manager.cc:4841] TS 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613): tablet 71328582fd1a4a29bfb142dd416c62d2 (table test-table [id=6f6dec2956d94f2c925fee6aba90da21]) successfully deleted
I20260927 21:33:10.971405 6144 ts_tablet_manager.cc:1939] T 71328582fd1a4a29bfb142dd416c62d2 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting consensus metadata
I20260927 21:33:10.972239 5544 catalog_manager.cc:4841] TS f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205): tablet 71328582fd1a4a29bfb142dd416c62d2 (table test-table [id=6f6dec2956d94f2c925fee6aba90da21]) successfully deleted
Sep 27 21:33:10 dist-test-slave-pqr8 krb5kdc[5515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:33:10 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:11.005232 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:10.989464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.3.186.254:41211)
0927 21:33:10.989937 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:10.989941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:10.990104 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:33:10.990234 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:10.990237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:10.990288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:10.990367 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:10.991446 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:10.992545 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:10.993753 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:10.994134 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:10.994237 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:11.000691 (+ 6454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:11.000696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:11.000699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:11.000725 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:11.003058 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:11.003699 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:11.003703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:11.003705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:11.003819 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:11.004210 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:11.004213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:11.004215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:11.004444 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:11.004620 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:11.004833 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:33:11.004980 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":260,"threads_started":1}
I20260927 21:33:11.008285 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:11.008911 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:11.014887 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 9b478d26fa454f9b8cd61089c6089412 (DEFAULT_TABLE table=test-table [id=3636e3a28bc745dba0393417a4ac0d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:11.014967 5836 tablet_service.cc:1467] Processing CreateTablet for tablet 9b478d26fa454f9b8cd61089c6089412 (DEFAULT_TABLE table=test-table [id=3636e3a28bc745dba0393417a4ac0d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:11.015340 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b478d26fa454f9b8cd61089c6089412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:11.015272 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 9b478d26fa454f9b8cd61089c6089412 (DEFAULT_TABLE table=test-table [id=3636e3a28bc745dba0393417a4ac0d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:11.015362 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b478d26fa454f9b8cd61089c6089412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:11.015605 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b478d26fa454f9b8cd61089c6089412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:11.020293 6155 tablet_bootstrap.cc:492] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Bootstrap starting.
I20260927 21:33:11.021759 6156 tablet_bootstrap.cc:492] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Bootstrap starting.
I20260927 21:33:11.021873 6155 tablet_bootstrap.cc:654] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:11.023901 6156 tablet_bootstrap.cc:654] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:11.026444 6157 tablet_bootstrap.cc:492] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Bootstrap starting.
I20260927 21:33:11.027129 6155 tablet_bootstrap.cc:492] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: No bootstrap required, opened a new log
I20260927 21:33:11.027268 6155 ts_tablet_manager.cc:1397] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 21:33:11.027437 6156 tablet_bootstrap.cc:492] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: No bootstrap required, opened a new log
I20260927 21:33:11.027559 6156 ts_tablet_manager.cc:1397] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260927 21:33:11.027983 6155 raft_consensus.cc:348] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.028227 6155 raft_consensus.cc:374] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:11.028326 6155 raft_consensus.cc:729] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c642981a404976afcf528da52cb681, State: Initialized, Role: FOLLOWER
I20260927 21:33:11.028232 6156 raft_consensus.cc:348] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.028426 6156 raft_consensus.cc:374] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:11.028514 6156 raft_consensus.cc:729] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fed1090f2a40c9a88509ea574fa893, State: Initialized, Role: FOLLOWER
I20260927 21:33:11.028548 6155 consensus_queue.cc:260] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.028734 6156 consensus_queue.cc:260] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.029471 6156 ts_tablet_manager.cc:1428] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:33:11.030531 6155 ts_tablet_manager.cc:1428] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 21:33:11.030900 6157 tablet_bootstrap.cc:654] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:11.032647 6157 tablet_bootstrap.cc:492] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: No bootstrap required, opened a new log
I20260927 21:33:11.032781 6157 ts_tablet_manager.cc:1397] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 21:33:11.033354 6157 raft_consensus.cc:348] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.033551 6157 raft_consensus.cc:374] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:11.033644 6157 raft_consensus.cc:729] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Initialized, Role: FOLLOWER
I20260927 21:33:11.033887 6157 consensus_queue.cc:260] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.034381 6157 ts_tablet_manager.cc:1428] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:33:11.075407 6160 raft_consensus.cc:482] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:11.075575 6160 raft_consensus.cc:504] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.076160 6160 leader_election.cc:288] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:11.076628 5722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b478d26fa454f9b8cd61089c6089412" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" is_pre_election: true
I20260927 21:33:11.076701 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b478d26fa454f9b8cd61089c6089412" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893" is_pre_election: true
I20260927 21:33:11.076877 5722 raft_consensus.cc:2454] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 0.
I20260927 21:33:11.076951 5994 raft_consensus.cc:2454] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 0.
I20260927 21:33:11.077350 5787 leader_election.cc:302] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:11.077620 6160 raft_consensus.cc:2781] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:11.077732 6160 raft_consensus.cc:482] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:11.077824 6160 raft_consensus.cc:3037] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:11.079576 6160 raft_consensus.cc:504] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.080021 6160 leader_election.cc:288] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 election: Requested vote from peers f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:11.080343 5722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b478d26fa454f9b8cd61089c6089412" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a"
I20260927 21:33:11.080513 5722 raft_consensus.cc:3037] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:11.080488 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b478d26fa454f9b8cd61089c6089412" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893"
I20260927 21:33:11.080677 5994 raft_consensus.cc:3037] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:11.082442 5722 raft_consensus.cc:2454] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 1.
I20260927 21:33:11.082459 5994 raft_consensus.cc:2454] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 1.
I20260927 21:33:11.082746 5787 leader_election.cc:302] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, 97fed1090f2a40c9a88509ea574fa893; no voters:
I20260927 21:33:11.083020 6160 raft_consensus.cc:2781] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:11.083469 6160 raft_consensus.cc:686] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 1 LEADER]: Becoming Leader. State: Replica: 38c642981a404976afcf528da52cb681, State: Running, Role: LEADER
I20260927 21:33:11.083797 6160 consensus_queue.cc:237] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.086107 5558 catalog_manager.cc:5495] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c642981a404976afcf528da52cb681 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "38c642981a404976afcf528da52cb681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:11.115432 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:11.110485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.3.186.194:40717)
0927 21:33:11.111044 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:11.111048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:11.111064 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:33:11.111109 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:11.111112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:11.111162 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:11.111265 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:11.111682 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:11.112417 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:11.113715 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:11.114143 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:11.114270 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:11.114691 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:11.114790 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:11.114884 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:11.115030 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:33:11.115099 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":326,"threads_started":1}
I20260927 21:33:11.122725 5994 raft_consensus.cc:1261] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Refusing update from remote peer 38c642981a404976afcf528da52cb681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:11.122781 5722 raft_consensus.cc:1261] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Refusing update from remote peer 38c642981a404976afcf528da52cb681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:11.123263 6160 consensus_queue.cc:1035] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:11.123528 6163 consensus_queue.cc:1035] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:11.164178 5558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:11.164405 5558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:11.166818 5558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Sending DeleteTablet for 3 replicas of tablet 9b478d26fa454f9b8cd61089c6089412
I20260927 21:33:11.167582 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b478d26fa454f9b8cd61089c6089412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:11 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:44026
I20260927 21:33:11.167557 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b478d26fa454f9b8cd61089c6089412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:11 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260927 21:33:11.167629 5836 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b478d26fa454f9b8cd61089c6089412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:11 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:52970
I20260927 21:33:11.167822 6144 tablet_replica.cc:330] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: stopping tablet replica
I20260927 21:33:11.167822 6145 tablet_replica.cc:330] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: stopping tablet replica
I20260927 21:33:11.168030 6146 tablet_replica.cc:330] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: stopping tablet replica
I20260927 21:33:11.168030 6144 raft_consensus.cc:2229] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:11.168030 6145 raft_consensus.cc:2229] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:11.168198 6144 raft_consensus.cc:2258] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:11.168243 6146 raft_consensus.cc:2229] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:11.168339 6145 raft_consensus.cc:2258] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:11.168848 6146 raft_consensus.cc:2258] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:11.169225 6144 ts_tablet_manager.cc:1905] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:11.169338 5558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:33:11.169816 6146 ts_tablet_manager.cc:1905] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:33:11.169899 5558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:11.170017 6145 ts_tablet_manager.cc:1905] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:11.175776 6146 ts_tablet_manager.cc:1918] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:11.175879 6146 log.cc:1196] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/9b478d26fa454f9b8cd61089c6089412
I20260927 21:33:11.176170 6146 ts_tablet_manager.cc:1939] T 9b478d26fa454f9b8cd61089c6089412 P 97fed1090f2a40c9a88509ea574fa893: Deleting consensus metadata
I20260927 21:33:11.176290 6145 ts_tablet_manager.cc:1918] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:11.176820 5696 tablet_service.cc:1467] Processing CreateTablet for tablet ec5f10cc7ab147259e8a7da030847870 (DEFAULT_TABLE table=test-table [id=3068fd135d8243fbb84d266ad37a8a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:11.177151 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5f10cc7ab147259e8a7da030847870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:11.177151 5975 tablet_service.cc:1467] Processing CreateTablet for tablet ec5f10cc7ab147259e8a7da030847870 (DEFAULT_TABLE table=test-table [id=3068fd135d8243fbb84d266ad37a8a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:11.177421 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5f10cc7ab147259e8a7da030847870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:11.178138 6144 ts_tablet_manager.cc:1918] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:11.178293 6144 log.cc:1196] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/9b478d26fa454f9b8cd61089c6089412
I20260927 21:33:11.178690 6144 ts_tablet_manager.cc:1939] T 9b478d26fa454f9b8cd61089c6089412 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting consensus metadata
I20260927 21:33:11.179596 5544 catalog_manager.cc:4841] TS f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205): tablet 9b478d26fa454f9b8cd61089c6089412 (table test-table [id=3636e3a28bc745dba0393417a4ac0d9b]) successfully deleted
I20260927 21:33:11.181675 6156 tablet_bootstrap.cc:492] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Bootstrap starting.
I20260927 21:33:11.181975 5544 catalog_manager.cc:4841] TS 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613): tablet 9b478d26fa454f9b8cd61089c6089412 (table test-table [id=3636e3a28bc745dba0393417a4ac0d9b]) successfully deleted
I20260927 21:33:11.178017 5836 tablet_service.cc:1467] Processing CreateTablet for tablet ec5f10cc7ab147259e8a7da030847870 (DEFAULT_TABLE table=test-table [id=3068fd135d8243fbb84d266ad37a8a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:11.182588 6157 tablet_bootstrap.cc:492] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Bootstrap starting.
I20260927 21:33:11.182793 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5f10cc7ab147259e8a7da030847870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:11.183066 6156 tablet_bootstrap.cc:654] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:11.184202 6157 tablet_bootstrap.cc:654] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:11.185191 6156 tablet_bootstrap.cc:492] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: No bootstrap required, opened a new log
I20260927 21:33:11.185367 6156 ts_tablet_manager.cc:1397] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:33:11.176508 6145 log.cc:1196] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/9b478d26fa454f9b8cd61089c6089412
I20260927 21:33:11.186028 6145 ts_tablet_manager.cc:1939] T 9b478d26fa454f9b8cd61089c6089412 P 38c642981a404976afcf528da52cb681: Deleting consensus metadata
I20260927 21:33:11.186282 6156 raft_consensus.cc:348] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.186655 6156 raft_consensus.cc:374] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:11.186797 6156 raft_consensus.cc:729] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fed1090f2a40c9a88509ea574fa893, State: Initialized, Role: FOLLOWER
I20260927 21:33:11.187110 6156 consensus_queue.cc:260] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.187796 6156 ts_tablet_manager.cc:1428] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:11.188596 6155 tablet_bootstrap.cc:492] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Bootstrap starting.
I20260927 21:33:11.188633 5545 catalog_manager.cc:4841] TS 38c642981a404976afcf528da52cb681 (127.3.186.194:40717): tablet 9b478d26fa454f9b8cd61089c6089412 (table test-table [id=3636e3a28bc745dba0393417a4ac0d9b]) successfully deleted
I20260927 21:33:11.190145 6157 tablet_bootstrap.cc:492] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: No bootstrap required, opened a new log
I20260927 21:33:11.190326 6157 ts_tablet_manager.cc:1397] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 21:33:11.190691 6155 tablet_bootstrap.cc:654] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:11.191224 6157 raft_consensus.cc:348] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.191547 6157 raft_consensus.cc:374] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:11.191672 6157 raft_consensus.cc:729] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4269b1b258842f9a2c47b69dcb4ff6a, State: Initialized, Role: FOLLOWER
I20260927 21:33:11.191977 6157 consensus_queue.cc:260] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.192405 6157 ts_tablet_manager.cc:1428] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 21:33:11.193164 6155 tablet_bootstrap.cc:492] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: No bootstrap required, opened a new log
I20260927 21:33:11.193325 6155 ts_tablet_manager.cc:1397] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 21:33:11.193962 6155 raft_consensus.cc:348] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.194173 6155 raft_consensus.cc:374] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:11.194267 6155 raft_consensus.cc:729] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c642981a404976afcf528da52cb681, State: Initialized, Role: FOLLOWER
I20260927 21:33:11.194489 6155 consensus_queue.cc:260] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.195022 6155 ts_tablet_manager.cc:1428] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:33:11.315979 6160 raft_consensus.cc:482] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:11.316152 6160 raft_consensus.cc:504] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.316697 6160 leader_election.cc:288] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:11.317132 5722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5f10cc7ab147259e8a7da030847870" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" is_pre_election: true
I20260927 21:33:11.317265 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5f10cc7ab147259e8a7da030847870" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893" is_pre_election: true
I20260927 21:33:11.317394 5722 raft_consensus.cc:2454] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 0.
I20260927 21:33:11.317481 5994 raft_consensus.cc:2454] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 0.
I20260927 21:33:11.317955 5787 leader_election.cc:302] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, f4269b1b258842f9a2c47b69dcb4ff6a; no voters:
I20260927 21:33:11.318228 6160 raft_consensus.cc:2781] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:11.318336 6160 raft_consensus.cc:482] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:11.318470 6160 raft_consensus.cc:3037] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:11.320715 6160 raft_consensus.cc:504] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.321205 6160 leader_election.cc:288] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 election: Requested vote from peers f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205), 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613)
I20260927 21:33:11.321576 5722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5f10cc7ab147259e8a7da030847870" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a"
I20260927 21:33:11.321691 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5f10cc7ab147259e8a7da030847870" candidate_uuid: "38c642981a404976afcf528da52cb681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97fed1090f2a40c9a88509ea574fa893"
I20260927 21:33:11.321774 5722 raft_consensus.cc:3037] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:11.321898 5994 raft_consensus.cc:3037] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:11.323616 5722 raft_consensus.cc:2454] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 1.
I20260927 21:33:11.323678 5994 raft_consensus.cc:2454] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c642981a404976afcf528da52cb681 in term 1.
I20260927 21:33:11.324038 5787 leader_election.cc:302] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c642981a404976afcf528da52cb681, 97fed1090f2a40c9a88509ea574fa893; no voters:
I20260927 21:33:11.324299 6160 raft_consensus.cc:2781] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:11.324438 6160 raft_consensus.cc:686] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 1 LEADER]: Becoming Leader. State: Replica: 38c642981a404976afcf528da52cb681, State: Running, Role: LEADER
I20260927 21:33:11.324681 6160 consensus_queue.cc:237] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } }
I20260927 21:33:11.327011 5560 catalog_manager.cc:5495] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c642981a404976afcf528da52cb681 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "38c642981a404976afcf528da52cb681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c642981a404976afcf528da52cb681" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:11.348316 5994 raft_consensus.cc:1261] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Refusing update from remote peer 38c642981a404976afcf528da52cb681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:11.348506 5722 raft_consensus.cc:1261] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Refusing update from remote peer 38c642981a404976afcf528da52cb681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:11.348860 6160 consensus_queue.cc:1035] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97fed1090f2a40c9a88509ea574fa893" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:11.349090 6163 consensus_queue.cc:1035] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4269b1b258842f9a2c47b69dcb4ff6a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:33:11.364900 6128 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:33:12.935787 5560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:12.935972 5560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:12.939462 5560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f4ed691c144f62b3be6299b9a2ae8d: Sending DeleteTablet for 3 replicas of tablet ec5f10cc7ab147259e8a7da030847870
I20260927 21:33:12.940162 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5f10cc7ab147259e8a7da030847870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:12 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260927 21:33:12.940294 5836 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5f10cc7ab147259e8a7da030847870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:12 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:52970
I20260927 21:33:12.940359 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5f10cc7ab147259e8a7da030847870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:12 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:44026
I20260927 21:33:12.940809 6198 tablet_replica.cc:330] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: stopping tablet replica
I20260927 21:33:12.940997 6198 raft_consensus.cc:2229] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:12.941217 6198 raft_consensus.cc:2258] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:12.941332 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5631
I20260927 21:33:12.942545 6198 ts_tablet_manager.cc:1905] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:12.951159 6198 ts_tablet_manager.cc:1918] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:33:12.951418 6198 log.cc:1196] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/ec5f10cc7ab147259e8a7da030847870
I20260927 21:33:12.952059 6198 ts_tablet_manager.cc:1939] T ec5f10cc7ab147259e8a7da030847870 P f4269b1b258842f9a2c47b69dcb4ff6a: Deleting consensus metadata
I20260927 21:33:12.953094 6200 tablet_replica.cc:330] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: stopping tablet replica
I20260927 21:33:12.953349 6200 raft_consensus.cc:2229] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:12.953356 5544 catalog_manager.cc:4841] TS f4269b1b258842f9a2c47b69dcb4ff6a (127.3.186.193:44205): tablet ec5f10cc7ab147259e8a7da030847870 (table test-table [id=3068fd135d8243fbb84d266ad37a8a67]) successfully deleted
I20260927 21:33:12.953945 6199 tablet_replica.cc:330] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: stopping tablet replica
I20260927 21:33:12.954217 6200 raft_consensus.cc:2258] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:12.954221 6199 raft_consensus.cc:2229] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:12.954596 6199 raft_consensus.cc:2258] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:12.955209 6200 ts_tablet_manager.cc:1905] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:12.955965 6199 ts_tablet_manager.cc:1905] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:12.961218 6199 ts_tablet_manager.cc:1918] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:33:12.961365 6199 log.cc:1196] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/ec5f10cc7ab147259e8a7da030847870
I20260927 21:33:12.961416 6200 ts_tablet_manager.cc:1918] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:33:12.961622 6200 log.cc:1196] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/ec5f10cc7ab147259e8a7da030847870
I20260927 21:33:12.961803 6199 ts_tablet_manager.cc:1939] T ec5f10cc7ab147259e8a7da030847870 P 38c642981a404976afcf528da52cb681: Deleting consensus metadata
I20260927 21:33:12.962123 6200 ts_tablet_manager.cc:1939] T ec5f10cc7ab147259e8a7da030847870 P 97fed1090f2a40c9a88509ea574fa893: Deleting consensus metadata
I20260927 21:33:12.962713 5545 catalog_manager.cc:4841] TS 38c642981a404976afcf528da52cb681 (127.3.186.194:40717): tablet ec5f10cc7ab147259e8a7da030847870 (table test-table [id=3068fd135d8243fbb84d266ad37a8a67]) successfully deleted
I20260927 21:33:12.963093 5544 catalog_manager.cc:4841] TS 97fed1090f2a40c9a88509ea574fa893 (127.3.186.195:42613): tablet ec5f10cc7ab147259e8a7da030847870 (table test-table [id=3068fd135d8243fbb84d266ad37a8a67]) successfully deleted
Sep 27 21:33:12 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:12.970286 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:12.943451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.3.186.193:44205)
0927 21:33:12.948931 (+ 5480us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:12.948939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:12.954835 (+ 5896us) server_negotiation.cc:407] Connection header received
0927 21:33:12.955004 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:12.955010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:12.955094 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:12.955228 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:12.957222 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:12.958349 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:12.960197 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:12.960847 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:12.961035 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:12.965408 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:12.965422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:12.965437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:12.965468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:12.968205 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:12.968735 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:12.968742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:12.968744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:12.968867 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:12.969150 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:12.969153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:12.969155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:12.969471 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:12.969749 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:12.969913 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 21:33:12.970050 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5302,"thread_start_us":5176,"threads_started":1}
W20260927 21:33:12.970858 5762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47216
W20260927 21:33:12.971431 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5631: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:12.971640 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5631
I20260927 21:33:12.996312 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5771
Sep 27 21:33:12 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:13.009795 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:12.996934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.3.186.194:40717)
0927 21:33:12.997437 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:12.997441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:12.997456 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:33:12.997496 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:12.997499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:12.997601 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:12.997688 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:12.998721 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:12.999381 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:13.000419 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:13.000780 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:13.000862 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:13.004595 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:13.004604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:13.004617 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:13.004644 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:13.007401 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:13.008152 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:13.008166 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:13.008168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:13.008274 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:13.008645 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:13.008650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:13.008654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:13.008969 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:13.009170 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:13.009383 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:33:13.009578 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":294,"threads_started":1}
W20260927 21:33:13.010412 5901 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52998
W20260927 21:33:13.010960 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5771: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:13.011058 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5771
I20260927 21:33:13.031791 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5910
Sep 27 21:33:13 dist-test-slave-pqr8 krb5kdc[5515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:13.044557 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:13.032365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.3.186.195:42613)
0927 21:33:13.032801 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:13.032805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:13.032819 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:33:13.032862 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:13.032865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:13.032915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:13.032996 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:13.034151 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:13.034862 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:13.035915 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:13.036278 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:13.036367 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:13.040041 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:13.040053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:13.040067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:13.040098 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:13.042465 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:13.043023 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:13.043029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:13.043031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:13.043147 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:13.043473 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:13.043476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:13.043478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:13.043772 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:13.043965 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:13.044180 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 21:33:13.044320 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":240,"threads_started":1}
W20260927 21:33:13.045143 6040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44036
W20260927 21:33:13.045812 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5910: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:13.045976 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5910
I20260927 21:33:13.062736 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 5528
I20260927 21:33:13.063714 5600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:13.173745 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 5528
2026-09-27T21:33:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8100 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:13 dist-test-slave-pqr8 krb5kdc[6208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:13 dist-test-slave-pqr8 krb5kdc[6208](info): set up 2 sockets
Sep 27 21:33:13 dist-test-slave-pqr8 krb5kdc[6208](info): commencing operation
krb5kdc: starting...
W20260927 21:33:14.225719 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:14 dist-test-slave-pqr8 krb5kdc[6208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:14.386567 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46481
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:14.681670 6221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:14.681888 6221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:14.682019 6221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:14.689368 6221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:14.689450 6221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:14.689549 6221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:14.689657 6221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:14.689740 6221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:14.698647 6221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:14.699743 6221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:14.700891 6221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:14.711726 6227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:14.711748 6226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:14.712213 6229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:14.712318 6221 server_base.cc:1034] running on GCE node
I20260927 21:33:14.712992 6221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:14.714238 6221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:14.715469 6221 hybrid_clock.cc:648] HybridClock initialized: now 1790544794715435 us; error 46 us; skew 500 ppm
I20260927 21:33:14.715765 6221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:14 dist-test-slave-pqr8 krb5kdc[6208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:14.718729 6221 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:14.720016 6221 webserver.cc:460] Webserver started at http://127.3.186.254:37677/ using document root <none> and password file <none>
I20260927 21:33:14.720610 6221 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:14.720741 6221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:14.721026 6221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:14.723742 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "2e417b3df35144e2867531deefb01a87"
format_stamp: "Formatted at 2026-09-27 21:33:14 on dist-test-slave-pqr8"
server_key: "e0bbde6899bdd1ea86c51d8cd3c8d12e"
server_key_iv: "18815ed9ced4cfb1dfc9d1263c0174da"
server_key_version: "encryptionkey@0"
I20260927 21:33:14.724476 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "2e417b3df35144e2867531deefb01a87"
format_stamp: "Formatted at 2026-09-27 21:33:14 on dist-test-slave-pqr8"
server_key: "e0bbde6899bdd1ea86c51d8cd3c8d12e"
server_key_iv: "18815ed9ced4cfb1dfc9d1263c0174da"
server_key_version: "encryptionkey@0"
I20260927 21:33:14.728937 6221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 21:33:14.731971 6236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:14.733027 6221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:14.733208 6221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "2e417b3df35144e2867531deefb01a87"
format_stamp: "Formatted at 2026-09-27 21:33:14 on dist-test-slave-pqr8"
server_key: "e0bbde6899bdd1ea86c51d8cd3c8d12e"
server_key_iv: "18815ed9ced4cfb1dfc9d1263c0174da"
server_key_version: "encryptionkey@0"
I20260927 21:33:14.733384 6221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:14.760140 6221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:14.764130 6221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:14.764405 6221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:14.781075 6221 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:46481
I20260927 21:33:14.781101 6287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:46481 every 8 connection(s)
I20260927 21:33:14.782351 6221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:14.785686 6288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:14.789978 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6221
I20260927 21:33:14.790189 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:14.790936 3819 external_mini_cluster.cc:1428] Setting key ca91f442b397fbc0acef37a6f9e2fb04
I20260927 21:33:14.796445 6288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87: Bootstrap starting.
I20260927 21:33:14.799701 6288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:14.800765 6288 log.cc:824] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:14.803792 6288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87: No bootstrap required, opened a new log
Sep 27 21:33:14 dist-test-slave-pqr8 krb5kdc[6208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:14.810844 6288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e417b3df35144e2867531deefb01a87" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46481 } }
I20260927 21:33:14.811252 6288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:14.811373 6288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e417b3df35144e2867531deefb01a87, State: Initialized, Role: FOLLOWER
I20260927 21:33:14.811872 6288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e417b3df35144e2867531deefb01a87" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46481 } }
I20260927 21:33:14.812104 6288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:14.812252 6288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:14.812291 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:14.792645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.3.186.254:46481)
0927 21:33:14.793337 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:14.793348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:14.793388 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:33:14.795064 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:14.795082 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:14.795412 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:14.795691 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:14.797423 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:14.798699 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:14.800382 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:14.801031 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:14.801192 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:14.805470 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:14.805477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:14.805492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:14.805569 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:14.808437 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:14.809082 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:14.809087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:14.809092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:14.809247 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:14.809627 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:14.809631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:14.809636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:14.810086 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:14.810381 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:14.810766 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 21:33:14.811134 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":307,"threads_started":1}
I20260927 21:33:14.812467 6288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:14.814934 6288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e417b3df35144e2867531deefb01a87" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46481 } }
I20260927 21:33:14.815409 6288 leader_election.cc:302] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e417b3df35144e2867531deefb01a87; no voters:
I20260927 21:33:14.815887 6288 leader_election.cc:288] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:14.816051 6293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:14.816759 6293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [term 1 LEADER]: Becoming Leader. State: Replica: 2e417b3df35144e2867531deefb01a87, State: Running, Role: LEADER
I20260927 21:33:14.817397 6293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e417b3df35144e2867531deefb01a87" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46481 } }
I20260927 21:33:14.817952 6288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:14.819936 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e417b3df35144e2867531deefb01a87. Latest consensus state: current_term: 1 leader_uuid: "2e417b3df35144e2867531deefb01a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e417b3df35144e2867531deefb01a87" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46481 } } }
I20260927 21:33:14.819983 6294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e417b3df35144e2867531deefb01a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e417b3df35144e2867531deefb01a87" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46481 } } }
I20260927 21:33:14.820319 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:14.820427 6294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:14.821157 6301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:14.828652 6301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:14.837471 6301 catalog_manager.cc:1347] Generated new cluster ID: ee4f2bf50d864087a61d312987a4f48c
I20260927 21:33:14.837598 6301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:14.848114 6301 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:14.849195 6301 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:14.857232 6301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e417b3df35144e2867531deefb01a87: Generated new TSK 0
I20260927 21:33:14.858038 6301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:14.904697 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46481
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:15.205865 6314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:15.206127 6314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:15.206287 6314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:15.212773 6314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:15.212899 6314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:15.213063 6314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:15.222208 6314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:15.223413 6314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:15.224565 6314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:15.236629 6322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:15.236672 6319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:15.236665 6320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:15.237331 6314 server_base.cc:1034] running on GCE node
I20260927 21:33:15.237928 6314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:15.238736 6314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:15.239979 6314 hybrid_clock.cc:648] HybridClock initialized: now 1790544795239918 us; error 49 us; skew 500 ppm
I20260927 21:33:15.240265 6314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:15 dist-test-slave-pqr8 krb5kdc[6208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:15.243041 6314 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:15.244359 6314 webserver.cc:460] Webserver started at http://127.3.186.193:34935/ using document root <none> and password file <none>
I20260927 21:33:15.244954 6314 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:15.245087 6314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:15.245368 6314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:15.247869 6314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "1a7958a8742f4d7b94bc79b88a7a3790"
format_stamp: "Formatted at 2026-09-27 21:33:15 on dist-test-slave-pqr8"
server_key: "057625f382a93c5092238c2e3255b586"
server_key_iv: "163e87756fbddcaa04966e9fe63b032d"
server_key_version: "encryptionkey@0"
I20260927 21:33:15.248534 6314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "1a7958a8742f4d7b94bc79b88a7a3790"
format_stamp: "Formatted at 2026-09-27 21:33:15 on dist-test-slave-pqr8"
server_key: "057625f382a93c5092238c2e3255b586"
server_key_iv: "163e87756fbddcaa04966e9fe63b032d"
server_key_version: "encryptionkey@0"
I20260927 21:33:15.252894 6314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 21:33:15.255908 6329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:15.257045 6314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:15.257216 6314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "1a7958a8742f4d7b94bc79b88a7a3790"
format_stamp: "Formatted at 2026-09-27 21:33:15 on dist-test-slave-pqr8"
server_key: "057625f382a93c5092238c2e3255b586"
server_key_iv: "163e87756fbddcaa04966e9fe63b032d"
server_key_version: "encryptionkey@0"
I20260927 21:33:15.257390 6314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:15.286203 6314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:15.291499 6314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:15.291895 6314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:15.292930 6314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:15.294370 6314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:15.294482 6314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:15.294595 6314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:15.294683 6314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:15.328053 6314 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:34743
I20260927 21:33:15.328101 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:34743 every 8 connection(s)
I20260927 21:33:15.329193 6314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:15.330166 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6314
I20260927 21:33:15.330387 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:15.330955 3819 external_mini_cluster.cc:1428] Setting key 2f5c0fd9a883167ab809a604187f9fac
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Sep 27 21:33:15 dist-test-slave-pqr8 krb5kdc[6208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:15.362102 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:15.333028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:43997 (local address 127.3.186.254:46481)
0927 21:33:15.333493 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:15.333502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:15.341015 (+ 7513us) server_negotiation.cc:407] Connection header received
0927 21:33:15.343262 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:15.343269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:15.343384 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:15.343494 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:15.345237 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:15.346490 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:15.348939 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:15.349625 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:15.349767 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:15.354634 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:15.354642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:15.354649 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:15.354689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:15.358390 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:15.359292 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:15.359299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:15.359305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:15.359474 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:15.359995 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:15.360003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:15.360009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:15.360399 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:15.360529 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:15.361397 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 21:33:15.361653 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":227,"threads_started":1}
I20260927 21:33:15.363080 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:15.332508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46481 (local address 127.3.186.193:43997)
0927 21:33:15.340296 (+ 7788us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:15.340350 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:15.342973 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:15.343715 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:15.343725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:15.344378 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:15.345048 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:15.345073 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:15.346666 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:15.346674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:15.348774 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:15.348789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:15.349844 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:15.349852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:15.350183 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:15.350908 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:15.350939 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:15.354407 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:15.358525 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:15.358534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:15.358549 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:15.359139 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:15.359608 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:15.359615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:15.359622 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:15.359860 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:15.360541 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:15.360550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:15.360955 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 21:33:15.361293 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 21:33:15.361835 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7514,"thread_start_us":7341,"threads_started":1}
I20260927 21:33:15.364583 6442 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46481
I20260927 21:33:15.364861 6442 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:15.365609 6442 heartbeater.cc:511] Master 127.3.186.254:46481 requested a full tablet report, sending...
I20260927 21:33:15.372412 6253 ts_manager.cc:194] Registered new tserver with Master: 1a7958a8742f4d7b94bc79b88a7a3790 (127.3.186.193:34743)
I20260927 21:33:15.374560 6253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:43997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
W20260927 21:33:16.272449 3819 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.186.194: real 0.938s user 0.006s sys 0.005s
I20260927 21:33:16.273123 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46481
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:33:16.377553 6442 heartbeater.cc:503] Master 127.3.186.254:46481 was elected leader, sending a full tablet report...
W20260927 21:33:16.595964 6448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:16.596206 6448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:16.596288 6448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:16.603726 6448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:16.603816 6448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:16.603912 6448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:16.613010 6448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:16.614141 6448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:16.615396 6448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:16.628060 6454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:16.628103 6453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:16.628124 6456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:16.629161 6448 server_base.cc:1034] running on GCE node
I20260927 21:33:16.629812 6448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:16.630671 6448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:16.632092 6448 hybrid_clock.cc:648] HybridClock initialized: now 1790544796631841 us; error 42 us; skew 500 ppm
I20260927 21:33:16.632424 6448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:16 dist-test-slave-pqr8 krb5kdc[6208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:16.639418 6448 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:16.640703 6448 webserver.cc:460] Webserver started at http://127.3.186.194:35199/ using document root <none> and password file <none>
I20260927 21:33:16.641295 6448 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:16.641427 6448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:16.641742 6448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:16.644328 6448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "20e468adafce45e2960cb6ee3d04c58b"
format_stamp: "Formatted at 2026-09-27 21:33:16 on dist-test-slave-pqr8"
server_key: "3f45b137039093ce69d9be3f74e3ff90"
server_key_iv: "f61a6e87d7703bca88289ac362fded1c"
server_key_version: "encryptionkey@0"
I20260927 21:33:16.645077 6448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "20e468adafce45e2960cb6ee3d04c58b"
format_stamp: "Formatted at 2026-09-27 21:33:16 on dist-test-slave-pqr8"
server_key: "3f45b137039093ce69d9be3f74e3ff90"
server_key_iv: "f61a6e87d7703bca88289ac362fded1c"
server_key_version: "encryptionkey@0"
I20260927 21:33:16.649502 6448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 21:33:16.652566 6463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:16.653788 6448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:16.653952 6448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "20e468adafce45e2960cb6ee3d04c58b"
format_stamp: "Formatted at 2026-09-27 21:33:16 on dist-test-slave-pqr8"
server_key: "3f45b137039093ce69d9be3f74e3ff90"
server_key_iv: "f61a6e87d7703bca88289ac362fded1c"
server_key_version: "encryptionkey@0"
I20260927 21:33:16.654125 6448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:16.674343 6448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:16.677824 6448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:16.678078 6448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:16.678908 6448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:16.680047 6448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:16.680150 6448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:16.680280 6448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:16.680349 6448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:16.712208 6448 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:33587
I20260927 21:33:16.712263 6575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:33587 every 8 connection(s)
I20260927 21:33:16.713796 6448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:16.718838 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6448
I20260927 21:33:16.719000 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:16.719529 3819 external_mini_cluster.cc:1428] Setting key 156f9b1d29bab9e443f394155ec9d5ba
Sep 27 21:33:16 dist-test-slave-pqr8 krb5kdc[6208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:16.733965 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:16.716294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:57991 (local address 127.3.186.254:46481)
0927 21:33:16.716689 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:16.716695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:16.717820 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 21:33:16.719744 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:16.719750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:16.719848 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:16.719963 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:16.721663 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:16.722500 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:16.724117 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:16.724517 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:16.724647 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:16.728532 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:16.728537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:16.728540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:16.728567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:16.731002 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:16.731858 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:16.731862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:16.731864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:16.731968 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:16.732411 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:16.732414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:16.732416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:16.732641 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:16.732739 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:16.733579 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 21:33:16.733721 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":185,"threads_started":1}
I20260927 21:33:16.735265 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:16.716911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46481 (local address 127.3.186.194:57991)
0927 21:33:16.717667 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:16.717709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:16.719452 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:16.720188 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:16.720199 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:16.720698 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:16.721395 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:16.721423 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:16.722669 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:16.722674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:16.723979 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:16.723990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:16.724718 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:16.724726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:16.724943 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:16.725634 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:16.725653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:16.728338 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:16.731128 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:16.731138 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:16.731155 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:16.731748 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:16.732074 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:16.732081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:16.732087 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:16.732305 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:16.732760 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:16.732771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:16.733172 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 21:33:16.733452 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 21:33:16.733992 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":296,"threads_started":1}
I20260927 21:33:16.736534 6576 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46481
I20260927 21:33:16.736848 6576 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:16.737566 6576 heartbeater.cc:511] Master 127.3.186.254:46481 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:33:16.738899 6253 ts_manager.cc:194] Registered new tserver with Master: 20e468adafce45e2960cb6ee3d04c58b (127.3.186.194:33587)
I20260927 21:33:16.739631 6253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:57991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
I20260927 21:33:17.741915 6576 heartbeater.cc:503] Master 127.3.186.254:46481 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
W20260927 21:33:17.787483 3819 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.186.195: real 1.065s user 0.003s sys 0.006s
I20260927 21:33:17.788370 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46481
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:18.117189 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:18.117481 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:18.117638 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:18.125914 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:18.126045 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:18.126242 6582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:18.135488 6582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:18.136638 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:18.138017 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:18.151176 6588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:18.151230 6587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:18.151294 6590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:18.151892 6582 server_base.cc:1034] running on GCE node
I20260927 21:33:18.152393 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:18.153132 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:18.154312 6582 hybrid_clock.cc:648] HybridClock initialized: now 1790544798154276 us; error 30 us; skew 500 ppm
I20260927 21:33:18.154641 6582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:18 dist-test-slave-pqr8 krb5kdc[6208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:18.157676 6582 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:18.158972 6582 webserver.cc:460] Webserver started at http://127.3.186.195:35983/ using document root <none> and password file <none>
I20260927 21:33:18.159632 6582 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:18.159754 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:18.160059 6582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:18.162910 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "cb2c1d3427914d21916d3f3b03a9630d"
format_stamp: "Formatted at 2026-09-27 21:33:18 on dist-test-slave-pqr8"
server_key: "f3cb68911e2bc9088cfdf77695ff0dc2"
server_key_iv: "f1d984d806d273c355d5b7b9508de20b"
server_key_version: "encryptionkey@0"
I20260927 21:33:18.163636 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "cb2c1d3427914d21916d3f3b03a9630d"
format_stamp: "Formatted at 2026-09-27 21:33:18 on dist-test-slave-pqr8"
server_key: "f3cb68911e2bc9088cfdf77695ff0dc2"
server_key_iv: "f1d984d806d273c355d5b7b9508de20b"
server_key_version: "encryptionkey@0"
I20260927 21:33:18.169725 6582 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 21:33:18.173043 6597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:18.174333 6582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:18.174525 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "cb2c1d3427914d21916d3f3b03a9630d"
format_stamp: "Formatted at 2026-09-27 21:33:18 on dist-test-slave-pqr8"
server_key: "f3cb68911e2bc9088cfdf77695ff0dc2"
server_key_iv: "f1d984d806d273c355d5b7b9508de20b"
server_key_version: "encryptionkey@0"
I20260927 21:33:18.174724 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:18.202160 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:18.206182 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:18.206449 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:18.207299 6582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:18.209195 6582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:18.209298 6582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:18.209444 6582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:18.209517 6582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:18.240916 6582 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:33107
I20260927 21:33:18.240989 6709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:33107 every 8 connection(s)
I20260927 21:33:18.242794 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:18.245895 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6582
I20260927 21:33:18.246129 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:18.246656 3819 external_mini_cluster.cc:1428] Setting key d9e142bb3401e322a6d7dd5cbfd527e8
Sep 27 21:33:18 dist-test-slave-pqr8 krb5kdc[6208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:18.261075 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:18.245000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:59013 (local address 127.3.186.254:46481)
0927 21:33:18.245452 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:18.245456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:18.246499 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 21:33:18.248557 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:18.248562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:18.248667 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:18.248780 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:18.250202 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:18.250942 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:18.252340 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:18.252731 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:18.252853 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:18.255912 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:18.255922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:18.255926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:18.255952 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:18.258346 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:18.258980 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:18.258985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:18.258989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:18.259099 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:18.259474 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:18.259479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:18.259482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:18.259711 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:18.259864 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:18.260493 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 21:33:18.260699 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
I20260927 21:33:18.262034 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:18.245410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46481 (local address 127.3.186.195:59013)
0927 21:33:18.246327 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:18.246380 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:18.248286 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:18.248994 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:18.249002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:18.249425 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:18.250050 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:18.250067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:18.251088 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:18.251092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:18.252208 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:18.252218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:18.252936 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:18.252944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:18.253143 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:18.253711 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:18.253731 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:18.255736 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:18.258471 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:18.258479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:18.258491 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:18.258868 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:18.259198 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:18.259204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:18.259208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:18.259362 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:18.259844 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:18.259850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:18.260124 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 21:33:18.260390 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 21:33:18.260890 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":309,"threads_started":1}
I20260927 21:33:18.263167 6710 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46481
I20260927 21:33:18.263485 6710 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:18.264129 6710 heartbeater.cc:511] Master 127.3.186.254:46481 requested a full tablet report, sending...
I20260927 21:33:18.265316 6253 ts_manager.cc:194] Registered new tserver with Master: cb2c1d3427914d21916d3f3b03a9630d (127.3.186.195:33107)
I20260927 21:33:18.266183 6253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:59013
I20260927 21:33:18.273468 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:33:18.301599 6712 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:18.299318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.3.186.254:46481)
0927 21:33:18.299491 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:18.299495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:18.300008 (+ 513us) server_negotiation.cc:407] Connection header received
0927 21:33:18.300135 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:18.300139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:18.300206 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:18.300289 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:18.301296 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49136: BlockingRecv error: recv got EOF from 127.0.0.1:49136 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:33:18.306337 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6314
W20260927 21:33:18.309022 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6314: Not authorized: Client connection negotiation failed: client connection to 127.3.186.193:34743: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:33:18.309181 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6314
I20260927 21:33:18.324378 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6448
W20260927 21:33:18.326586 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6448: Not authorized: Client connection negotiation failed: client connection to 127.3.186.194:33587: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:33:18.326767 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6448
I20260927 21:33:18.341163 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6582
W20260927 21:33:18.343915 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6582: Not authorized: Client connection negotiation failed: client connection to 127.3.186.195:33107: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:33:18.344039 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6582
I20260927 21:33:18.358809 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6221
I20260927 21:33:18.359835 6283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:18.462325 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6221
2026-09-27T21:33:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5287 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:20 dist-test-slave-pqr8 krb5kdc[6728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:20 dist-test-slave-pqr8 krb5kdc[6728](info): set up 2 sockets
Sep 27 21:33:20 dist-test-slave-pqr8 krb5kdc[6728](info): commencing operation
krb5kdc: starting...
W20260927 21:33:21.017800 3819 mini_kdc.cc:121] Time spent starting KDC: real 2.514s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:21 dist-test-slave-pqr8 krb5kdc[6728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:21.178727 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46873
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:46873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:21.478802 6741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:21.479080 6741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:21.479234 6741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:21.485929 6741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:21.486058 6741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:21.486155 6741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:21.486246 6741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:21.486337 6741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:21.495204 6741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:46873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:21.496291 6741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:21.497452 6741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:21.508355 6746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:21.508419 6747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:21.508401 6749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:21.509207 6741 server_base.cc:1034] running on GCE node
I20260927 21:33:21.509899 6741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:21.511039 6741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:21.512225 6741 hybrid_clock.cc:648] HybridClock initialized: now 1790544801512185 us; error 38 us; skew 500 ppm
I20260927 21:33:21.512533 6741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:21 dist-test-slave-pqr8 krb5kdc[6728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:21.515311 6741 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:21.516589 6741 webserver.cc:460] Webserver started at http://127.3.186.254:42513/ using document root <none> and password file <none>
I20260927 21:33:21.517124 6741 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:21.517215 6741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:21.517450 6741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:21.520120 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "b91193a9cf7241239a2446d7128b7dcc"
format_stamp: "Formatted at 2026-09-27 21:33:21 on dist-test-slave-pqr8"
server_key: "8c3f603ac30f696470f8042424d8709f"
server_key_iv: "3be72e26c00277a4ce8acded65078a71"
server_key_version: "encryptionkey@0"
I20260927 21:33:21.520809 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "b91193a9cf7241239a2446d7128b7dcc"
format_stamp: "Formatted at 2026-09-27 21:33:21 on dist-test-slave-pqr8"
server_key: "8c3f603ac30f696470f8042424d8709f"
server_key_iv: "3be72e26c00277a4ce8acded65078a71"
server_key_version: "encryptionkey@0"
I20260927 21:33:21.525240 6741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 21:33:21.528173 6756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:21.529328 6741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:21.529493 6741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "b91193a9cf7241239a2446d7128b7dcc"
format_stamp: "Formatted at 2026-09-27 21:33:21 on dist-test-slave-pqr8"
server_key: "8c3f603ac30f696470f8042424d8709f"
server_key_iv: "3be72e26c00277a4ce8acded65078a71"
server_key_version: "encryptionkey@0"
I20260927 21:33:21.529740 6741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:21.555398 6741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:21.558832 6741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:21.559077 6741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:21.575923 6741 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:46873
I20260927 21:33:21.575995 6807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:46873 every 8 connection(s)
I20260927 21:33:21.577096 6741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:21.580369 6808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:21.582131 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6741
I20260927 21:33:21.582324 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:21.582897 3819 external_mini_cluster.cc:1428] Setting key a6154a10e925434e5ad22e0e0ef25ab5
I20260927 21:33:21.592561 6808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc: Bootstrap starting.
I20260927 21:33:21.596040 6808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc: Neither blocks nor log segments found. Creating new log.
Sep 27 21:33:21 dist-test-slave-pqr8 krb5kdc[6728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:21.596999 6808 log.cc:824] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:21.600101 6808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc: No bootstrap required, opened a new log
I20260927 21:33:21.605675 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:21.584610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.3.186.254:46873)
0927 21:33:21.585412 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:21.585431 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:21.585480 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:33:21.587712 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:21.587738 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:21.588186 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:21.588575 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:21.589908 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:21.591776 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:21.592958 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:21.593939 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:21.594142 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:21.598135 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:21.598141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:21.598178 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:21.598213 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:21.601684 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:21.602245 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:21.602255 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:21.602264 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:21.602471 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:21.602787 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:21.602791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:21.602795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:21.603405 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:21.603691 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:21.604054 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 21:33:21.604422 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":346,"threads_started":1}
I20260927 21:33:21.609290 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:33:21.609297 6808 raft_consensus.cc:348] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91193a9cf7241239a2446d7128b7dcc" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46873 } }
I20260927 21:33:21.609719 6808 raft_consensus.cc:374] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:21.609817 6808 raft_consensus.cc:729] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91193a9cf7241239a2446d7128b7dcc, State: Initialized, Role: FOLLOWER
I20260927 21:33:21.610334 6808 consensus_queue.cc:260] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91193a9cf7241239a2446d7128b7dcc" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46873 } }
I20260927 21:33:21.610546 6808 raft_consensus.cc:388] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:21.610666 6808 raft_consensus.cc:482] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:21.610838 6808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:21.612718 6808 raft_consensus.cc:504] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91193a9cf7241239a2446d7128b7dcc" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46873 } }
I20260927 21:33:21.613164 6808 leader_election.cc:302] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b91193a9cf7241239a2446d7128b7dcc; no voters:
I20260927 21:33:21.613770 6808 leader_election.cc:288] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:21.613850 6813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:21.615151 6813 raft_consensus.cc:686] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [term 1 LEADER]: Becoming Leader. State: Replica: b91193a9cf7241239a2446d7128b7dcc, State: Running, Role: LEADER
I20260927 21:33:21.615720 6813 consensus_queue.cc:237] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91193a9cf7241239a2446d7128b7dcc" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46873 } }
I20260927 21:33:21.615962 6808 sys_catalog.cc:564] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:21.618479 6815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b91193a9cf7241239a2446d7128b7dcc. Latest consensus state: current_term: 1 leader_uuid: "b91193a9cf7241239a2446d7128b7dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91193a9cf7241239a2446d7128b7dcc" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46873 } } }
I20260927 21:33:21.618798 6815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:21.619385 6814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b91193a9cf7241239a2446d7128b7dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91193a9cf7241239a2446d7128b7dcc" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46873 } } }
I20260927 21:33:21.619681 6814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:21.622246 6822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:21.627529 6822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:21.637032 6822 catalog_manager.cc:1347] Generated new cluster ID: a088b8d6e4bb4d2f85cd47197f8faa9d
I20260927 21:33:21.637123 6822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:21.663460 6822 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:21.664537 6822 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:21.668053 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:21.655340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.3.186.254:46873)
0927 21:33:21.655482 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:21.655489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:21.656101 (+ 612us) server_negotiation.cc:407] Connection header received
0927 21:33:21.656239 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:21.656244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:21.656325 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:21.656424 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:21.657368 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:21.658448 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:21.660019 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:21.660627 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:21.660747 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:21.666528 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:21.666534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:21.666539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:21.667246 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:21.667323 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:21.667357 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:21.667464 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:21.667594 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:33:21.667708 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":45}
I20260927 21:33:21.670423 6822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b91193a9cf7241239a2446d7128b7dcc: Generated new TSK 0
I20260927 21:33:21.671068 6822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:33:21.677338 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6741
I20260927 21:33:21.678512 6803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:21.761369 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6741
2026-09-27T21:33:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3293 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:21 dist-test-slave-pqr8 krb5kdc[6842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:21 dist-test-slave-pqr8 krb5kdc[6842](info): set up 2 sockets
Sep 27 21:33:21 dist-test-slave-pqr8 krb5kdc[6842](info): commencing operation
krb5kdc: starting...
W20260927 21:33:22.797797 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:22 dist-test-slave-pqr8 krb5kdc[6842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:22.962517 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:36675
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:36675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:23.259714 6855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:23.259955 6855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:23.260098 6855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:23.266893 6855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:23.267005 6855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:23.267109 6855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:23.267207 6855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:23.267288 6855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:23.276155 6855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:36675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:36675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:23.277243 6855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:23.278457 6855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:23.289166 6860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:23.289189 6863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:23.289213 6861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:23.290108 6855 server_base.cc:1034] running on GCE node
I20260927 21:33:23.290799 6855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:23.292007 6855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:23.293234 6855 hybrid_clock.cc:648] HybridClock initialized: now 1790544803293200 us; error 42 us; skew 500 ppm
I20260927 21:33:23.293604 6855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:23 dist-test-slave-pqr8 krb5kdc[6842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:23.296541 6855 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:23.297886 6855 webserver.cc:460] Webserver started at http://127.3.186.254:43427/ using document root <none> and password file <none>
I20260927 21:33:23.298434 6855 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:23.298525 6855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:23.298779 6855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:23.301417 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "d933d351b0a740ce84a2ad0de0727176"
format_stamp: "Formatted at 2026-09-27 21:33:23 on dist-test-slave-pqr8"
server_key: "80b38caf287a63113bf9bfc71da34d00"
server_key_iv: "235f249dff0863d322c293de281f8d15"
server_key_version: "encryptionkey@0"
I20260927 21:33:23.302125 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "d933d351b0a740ce84a2ad0de0727176"
format_stamp: "Formatted at 2026-09-27 21:33:23 on dist-test-slave-pqr8"
server_key: "80b38caf287a63113bf9bfc71da34d00"
server_key_iv: "235f249dff0863d322c293de281f8d15"
server_key_version: "encryptionkey@0"
I20260927 21:33:23.306676 6855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:33:23.309680 6870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:23.310918 6855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:23.311180 6855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "d933d351b0a740ce84a2ad0de0727176"
format_stamp: "Formatted at 2026-09-27 21:33:23 on dist-test-slave-pqr8"
server_key: "80b38caf287a63113bf9bfc71da34d00"
server_key_iv: "235f249dff0863d322c293de281f8d15"
server_key_version: "encryptionkey@0"
I20260927 21:33:23.311374 6855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:23.340965 6855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:23.344504 6855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:23.344762 6855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:23.361181 6855 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:36675
I20260927 21:33:23.361217 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:36675 every 8 connection(s)
I20260927 21:33:23.362317 6855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:23.365437 6922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:23.365993 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6855
I20260927 21:33:23.366196 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:23.366928 3819 external_mini_cluster.cc:1428] Setting key aa99a6850250493b11d395ed3789672a
I20260927 21:33:23.378391 6922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176: Bootstrap starting.
Sep 27 21:33:23 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:23.381717 6922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:23.382797 6922 log.cc:824] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:23.385391 6922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176: No bootstrap required, opened a new log
I20260927 21:33:23.388423 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:23.368814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.3.186.254:36675)
0927 21:33:23.369609 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:23.369619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:23.369653 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:33:23.371289 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:23.371309 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:23.371652 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:23.371925 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:23.373427 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:23.374740 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:23.376497 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:23.377072 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:23.377187 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:23.381446 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:23.381453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:23.381465 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:23.381494 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:23.384754 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:23.385390 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:23.385402 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:23.385407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:23.385636 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:23.386005 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:23.386011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:23.386015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:23.386473 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:23.386781 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:23.387206 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 21:33:23.387564 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":345,"threads_started":1}
I20260927 21:33:23.392768 6922 raft_consensus.cc:348] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d933d351b0a740ce84a2ad0de0727176" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 36675 } }
I20260927 21:33:23.393092 6922 raft_consensus.cc:374] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:23.393191 6922 raft_consensus.cc:729] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d933d351b0a740ce84a2ad0de0727176, State: Initialized, Role: FOLLOWER
I20260927 21:33:23.393729 6922 consensus_queue.cc:260] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d933d351b0a740ce84a2ad0de0727176" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 36675 } }
I20260927 21:33:23.393946 6922 raft_consensus.cc:388] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:23.394089 6922 raft_consensus.cc:482] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:23.394240 6922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:23.396199 6922 raft_consensus.cc:504] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d933d351b0a740ce84a2ad0de0727176" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 36675 } }
I20260927 21:33:23.396718 6922 leader_election.cc:302] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d933d351b0a740ce84a2ad0de0727176; no voters:
I20260927 21:33:23.397286 6922 leader_election.cc:288] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:23.397370 6927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:23.398013 6927 raft_consensus.cc:686] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [term 1 LEADER]: Becoming Leader. State: Replica: d933d351b0a740ce84a2ad0de0727176, State: Running, Role: LEADER
I20260927 21:33:23.398553 6927 consensus_queue.cc:237] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d933d351b0a740ce84a2ad0de0727176" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 36675 } }
I20260927 21:33:23.399223 6922 sys_catalog.cc:564] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:23.401281 6929 sys_catalog.cc:455] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933d351b0a740ce84a2ad0de0727176. Latest consensus state: current_term: 1 leader_uuid: "d933d351b0a740ce84a2ad0de0727176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d933d351b0a740ce84a2ad0de0727176" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 36675 } } }
I20260927 21:33:23.401654 6929 sys_catalog.cc:458] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:23.402160 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d933d351b0a740ce84a2ad0de0727176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d933d351b0a740ce84a2ad0de0727176" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 36675 } } }
I20260927 21:33:23.405649 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:23.407075 6936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:23.412206 6936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:23.419870 6936 catalog_manager.cc:1347] Generated new cluster ID: 7043b3f0f46e42c7bfc4e6a449c1cb86
I20260927 21:33:23.419989 6936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:23.429150 6936 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:23.430399 6936 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:23.438717 6936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d933d351b0a740ce84a2ad0de0727176: Generated new TSK 0
I20260927 21:33:23.439318 6936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:23.482664 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:36675
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:23.789249 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:23.789551 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:23.789681 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:23.796774 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:23.796865 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:23.797027 6948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:23.806569 6948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:36675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:23.807582 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:23.808696 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:23.820737 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:23.820955 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:23.821131 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:23.821913 6948 server_base.cc:1034] running on GCE node
I20260927 21:33:23.822508 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:23.823361 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:23.824604 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790544803824572 us; error 45 us; skew 500 ppm
I20260927 21:33:23.824887 6948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:23 dist-test-slave-pqr8 krb5kdc[6842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:23.827883 6948 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:23.829290 6948 webserver.cc:460] Webserver started at http://127.3.186.193:41699/ using document root <none> and password file <none>
I20260927 21:33:23.829952 6948 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:23.830097 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:23.830381 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:23.832947 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "162e8e7e44874e41b896a405edbf9c9a"
format_stamp: "Formatted at 2026-09-27 21:33:23 on dist-test-slave-pqr8"
server_key: "ee81f6ab8a4ba20fc4bf7ba2019e2d56"
server_key_iv: "ccfd19e1289e9d15bdfd62cab5b745d2"
server_key_version: "encryptionkey@0"
I20260927 21:33:23.833735 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "162e8e7e44874e41b896a405edbf9c9a"
format_stamp: "Formatted at 2026-09-27 21:33:23 on dist-test-slave-pqr8"
server_key: "ee81f6ab8a4ba20fc4bf7ba2019e2d56"
server_key_iv: "ccfd19e1289e9d15bdfd62cab5b745d2"
server_key_version: "encryptionkey@0"
I20260927 21:33:23.838394 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:23.841464 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:23.842629 6948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:23.842790 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "162e8e7e44874e41b896a405edbf9c9a"
format_stamp: "Formatted at 2026-09-27 21:33:23 on dist-test-slave-pqr8"
server_key: "ee81f6ab8a4ba20fc4bf7ba2019e2d56"
server_key_iv: "ccfd19e1289e9d15bdfd62cab5b745d2"
server_key_version: "encryptionkey@0"
I20260927 21:33:23.842983 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:23.869038 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:23.872437 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:23.872704 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:23.873548 6948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:23.874897 6948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:23.874985 6948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:23.875110 6948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:23.875169 6948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:23.907464 6948 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:37117
I20260927 21:33:23.907537 7075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:37117 every 8 connection(s)
I20260927 21:33:23.908567 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:23.916746 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 6948
I20260927 21:33:23.916956 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:23.917701 3819 external_mini_cluster.cc:1428] Setting key c4abdc81a0618825ee9551882bb4077c
Sep 27 21:33:23 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:23.928706 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:23.910957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:37407 (local address 127.3.186.254:36675)
0927 21:33:23.911391 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:23.911396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:23.912144 (+ 748us) server_negotiation.cc:407] Connection header received
0927 21:33:23.914065 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:23.914071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:23.914153 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:23.914275 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:23.915754 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:23.916740 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:23.918460 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:23.918959 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:23.919065 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:23.922402 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:23.922411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:23.922420 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:23.922466 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:23.925763 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:23.926619 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:23.926623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:23.926625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:23.926735 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:23.927177 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:23.927180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:23.927182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:23.927430 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:23.927538 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:23.928306 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 21:33:23.928450 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":249,"threads_started":1}
I20260927 21:33:23.930104 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:23.911310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:36675 (local address 127.3.186.193:37407)
0927 21:33:23.911997 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:23.912037 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:23.913793 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:23.914559 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:23.914568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:23.915044 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:23.915586 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:23.915602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:23.916904 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:23.916916 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:23.918314 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:23.918329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:23.919168 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:23.919174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:23.919461 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:23.920156 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:23.920175 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:23.922188 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:23.925892 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:23.925900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:23.925914 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:23.926498 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:23.926835 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:23.926842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:23.926849 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:23.927068 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:23.927548 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:23.927566 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:23.927965 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 21:33:23.928225 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 21:33:23.928725 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260927 21:33:23.931423 7076 heartbeater.cc:347] Connected to a master server at 127.3.186.254:36675
I20260927 21:33:23.931702 7076 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:23.932343 7076 heartbeater.cc:511] Master 127.3.186.254:36675 requested a full tablet report, sending...
I20260927 21:33:23.934369 6887 ts_manager.cc:194] Registered new tserver with Master: 162e8e7e44874e41b896a405edbf9c9a (127.3.186.193:37117)
I20260927 21:33:23.935851 6887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:37407
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:33:23.958673 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:36675
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:24.270462 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:24.270735 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:24.270900 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:24.277588 7082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:24.277714 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:24.277880 7082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:24.287038 7082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:36675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:24.288158 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:24.289341 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:24.301414 7090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:24.301456 7088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:24.301471 7087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:24.302142 7082 server_base.cc:1034] running on GCE node
I20260927 21:33:24.302649 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:24.303499 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:24.304682 7082 hybrid_clock.cc:648] HybridClock initialized: now 1790544804304648 us; error 29 us; skew 500 ppm
I20260927 21:33:24.304961 7082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:24.307833 7082 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:24.309026 7082 webserver.cc:460] Webserver started at http://127.3.186.194:42293/ using document root <none> and password file <none>
I20260927 21:33:24.309649 7082 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:24.309744 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:24.309985 7082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:24.312561 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "d98aab93a7ca45c2bf40a962d802b807"
format_stamp: "Formatted at 2026-09-27 21:33:24 on dist-test-slave-pqr8"
server_key: "425274bf0818e7cf77f0d342a281b167"
server_key_iv: "cf8c13305bade2a719126ae5cf5133fe"
server_key_version: "encryptionkey@0"
I20260927 21:33:24.313225 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "d98aab93a7ca45c2bf40a962d802b807"
format_stamp: "Formatted at 2026-09-27 21:33:24 on dist-test-slave-pqr8"
server_key: "425274bf0818e7cf77f0d342a281b167"
server_key_iv: "cf8c13305bade2a719126ae5cf5133fe"
server_key_version: "encryptionkey@0"
I20260927 21:33:24.318264 7082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:33:24.321380 7097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:24.322715 7082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:33:24.322877 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "d98aab93a7ca45c2bf40a962d802b807"
format_stamp: "Formatted at 2026-09-27 21:33:24 on dist-test-slave-pqr8"
server_key: "425274bf0818e7cf77f0d342a281b167"
server_key_iv: "cf8c13305bade2a719126ae5cf5133fe"
server_key_version: "encryptionkey@0"
I20260927 21:33:24.323056 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:24.368352 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:24.372025 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:24.372315 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:24.373222 7082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:24.374471 7082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:24.374540 7082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:24.374691 7082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:24.374743 7082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:24.406656 7082 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:34619
I20260927 21:33:24.406692 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:34619 every 8 connection(s)
I20260927 21:33:24.407780 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:24.413436 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7082
I20260927 21:33:24.413627 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:24.414163 3819 external_mini_cluster.cc:1428] Setting key 68785e952232cde55ddaf96888ab9b4d
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:24.429836 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.410282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:59027 (local address 127.3.186.254:36675)
0927 21:33:24.410484 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:24.410490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:24.411560 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 21:33:24.413303 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:24.413306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:24.413365 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:24.413451 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:24.415256 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.416296 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.418796 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.419171 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.419259 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.423107 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:24.423116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:24.423124 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:24.423160 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:24.426732 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.427419 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.427428 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.427434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.427592 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.428022 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.428029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.428035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.428370 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:24.428527 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:24.429137 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 21:33:24.429384 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:33:24.430549 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.410772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:36675 (local address 127.3.186.194:59027)
0927 21:33:24.411404 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:24.411439 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:24.412976 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:24.413744 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:24.413754 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:24.414479 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:24.415107 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:24.415124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.416462 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.416467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:24.418654 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:24.418671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.419398 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.419406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:24.419739 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.420406 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:24.420438 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:24.422900 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:24.426879 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:24.426888 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:24.426903 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:24.427287 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:24.427710 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:24.427713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:24.427715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:24.427888 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:24.428482 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:24.428487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:24.428777 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 21:33:24.428986 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 21:33:24.429313 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":249,"threads_started":1}
I20260927 21:33:24.431854 7210 heartbeater.cc:347] Connected to a master server at 127.3.186.254:36675
I20260927 21:33:24.432122 7210 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:24.432758 7210 heartbeater.cc:511] Master 127.3.186.254:36675 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:33:24.434024 6887 ts_manager.cc:194] Registered new tserver with Master: d98aab93a7ca45c2bf40a962d802b807 (127.3.186.194:34619)
I20260927 21:33:24.434932 6887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:59027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:33:24.455741 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:36675
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:24.750504 7215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:24.750730 7215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:24.750811 7215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:24.757249 7215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:24.757328 7215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:24.757421 7215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:24.765750 7215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:36675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:24.766844 7215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:24.767997 7215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:24.779696 7220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:24.779923 7223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:24.780067 7221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:24.780848 7215 server_base.cc:1034] running on GCE node
I20260927 21:33:24.781412 7215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:24.782336 7215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:24.783519 7215 hybrid_clock.cc:648] HybridClock initialized: now 1790544804783482 us; error 36 us; skew 500 ppm
I20260927 21:33:24.783805 7215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:24.786777 7215 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:24.788134 7215 webserver.cc:460] Webserver started at http://127.3.186.195:41147/ using document root <none> and password file <none>
I20260927 21:33:24.788764 7215 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:24.788900 7215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:24.789178 7215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:24.791831 7215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "6fb85cd3b8554441a4cdf05e887165ad"
format_stamp: "Formatted at 2026-09-27 21:33:24 on dist-test-slave-pqr8"
server_key: "7d3d4f40d6193d3523a915bad9fff1d2"
server_key_iv: "4cb642122dcaf2ccd7e83edf73ff2ea0"
server_key_version: "encryptionkey@0"
I20260927 21:33:24.792559 7215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "6fb85cd3b8554441a4cdf05e887165ad"
format_stamp: "Formatted at 2026-09-27 21:33:24 on dist-test-slave-pqr8"
server_key: "7d3d4f40d6193d3523a915bad9fff1d2"
server_key_iv: "4cb642122dcaf2ccd7e83edf73ff2ea0"
server_key_version: "encryptionkey@0"
I20260927 21:33:24.797222 7215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 21:33:24.800490 7230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:24.801786 7215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:24.801954 7215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "6fb85cd3b8554441a4cdf05e887165ad"
format_stamp: "Formatted at 2026-09-27 21:33:24 on dist-test-slave-pqr8"
server_key: "7d3d4f40d6193d3523a915bad9fff1d2"
server_key_iv: "4cb642122dcaf2ccd7e83edf73ff2ea0"
server_key_version: "encryptionkey@0"
I20260927 21:33:24.802153 7215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:24.817171 7215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:24.821135 7215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:24.821415 7215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:24.822230 7215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:24.823448 7215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:24.823518 7215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:24.823678 7215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:24.823728 7215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:24.853473 7215 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:45055
I20260927 21:33:24.853580 7342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:45055 every 8 connection(s)
I20260927 21:33:24.854581 7215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:24.859113 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7215
I20260927 21:33:24.859299 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:24.859711 3819 external_mini_cluster.cc:1428] Setting key 5717656afc33171f09833f90f3d5dbf8
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:24.872072 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.856803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:41393 (local address 127.3.186.254:36675)
0927 21:33:24.857011 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:24.857017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:24.858182 (+ 1165us) server_negotiation.cc:407] Connection header received
0927 21:33:24.859922 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:24.859929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:24.860011 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:24.860112 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:24.861579 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.862373 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.863746 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.864151 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.864293 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.867334 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:24.867338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:24.867341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:24.867368 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:24.869470 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.870135 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.870140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.870144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.870268 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.870631 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.870636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.870639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.870875 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:24.871010 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:24.871567 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 21:33:24.871764 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:33:24.872800 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.857241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:36675 (local address 127.3.186.195:41393)
0927 21:33:24.858008 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:24.858041 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:24.859654 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:24.860363 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:24.860374 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:24.860850 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:24.861420 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:24.861437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.862525 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.862528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:24.863605 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:24.863615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.864369 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.864378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:24.864610 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.865107 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:24.865126 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:24.867128 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:24.869649 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:24.869657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:24.869669 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:24.870017 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:24.870367 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:24.870371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:24.870375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:24.870527 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:24.870988 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:24.870995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:24.871248 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 21:33:24.871472 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 21:33:24.871878 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":326,"threads_started":1}
I20260927 21:33:24.874051 7343 heartbeater.cc:347] Connected to a master server at 127.3.186.254:36675
I20260927 21:33:24.874312 7343 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:24.874918 7343 heartbeater.cc:511] Master 127.3.186.254:36675 requested a full tablet report, sending...
I20260927 21:33:24.876120 6887 ts_manager.cc:194] Registered new tserver with Master: 6fb85cd3b8554441a4cdf05e887165ad (127.3.186.195:45055)
I20260927 21:33:24.876881 6887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:41393
I20260927 21:33:24.886099 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:24.929265 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.916903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.3.186.254:36675)
0927 21:33:24.917106 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:24.917114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:24.917425 (+ 311us) server_negotiation.cc:407] Connection header received
0927 21:33:24.917594 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:24.917598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:24.917652 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:24.917747 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:24.918944 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.919624 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.920778 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.921141 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.921223 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.924609 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:24.924613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:24.924616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:24.924642 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:24.927297 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.927829 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.927833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.927835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.927951 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.928234 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.928238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.928241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.928502 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:24.928670 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:24.928839 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:33:24.928978 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 21:33:24.929785 6887 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43126
I20260927 21:33:24.931136 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6948
I20260927 21:33:24.938594 7076 heartbeater.cc:503] Master 127.3.186.254:36675 was elected leader, sending a full tablet report...
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:24.945247 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.932207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.3.186.193:37117)
0927 21:33:24.932715 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:24.932721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:24.932752 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:33:24.932799 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:24.932804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:24.932978 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:24.933086 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:24.934173 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.935112 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.936174 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.936665 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.936774 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.940407 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:24.940416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:24.940427 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:24.940475 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:24.943112 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.943623 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.943628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.943630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.943736 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.944020 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.944023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.944025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.944341 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:24.944539 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:24.944704 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:33:24.944877 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
W20260927 21:33:24.946357 7071 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36064
W20260927 21:33:24.947357 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:24.947551 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6948
I20260927 21:33:24.962317 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7082
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:24.975764 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.962965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.3.186.194:34619)
0927 21:33:24.963403 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:24.963408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:24.963428 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:33:24.963476 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:24.963481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:24.963635 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:24.963750 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:24.964734 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.965758 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.966894 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.967362 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.967448 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:24.970891 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:24.970900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:24.970911 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:24.970936 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:24.973444 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.974015 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.974026 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.974030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.974167 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:24.974476 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:24.974480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:24.974483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:24.974828 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:24.975032 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:24.975193 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:33:24.975356 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
W20260927 21:33:24.976857 7205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46472
W20260927 21:33:24.977874 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7082: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:24.977999 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7082
I20260927 21:33:24.992260 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7215
Sep 27 21:33:24 dist-test-slave-pqr8 krb5kdc[6842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:25.006798 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:24.992970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.3.186.195:45055)
0927 21:33:24.993465 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:24.993470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:24.993493 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:33:24.993591 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:24.993596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:24.993749 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:24.993892 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:24.995000 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.996004 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.997139 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:24.997732 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:24.997873 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:25.001642 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:25.001651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:25.001660 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:25.001714 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:25.004170 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:25.004784 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:25.004789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:25.004791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:25.004894 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:25.005244 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:25.005254 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:25.005256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:25.005765 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:25.005970 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:25.006260 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 21:33:25.006430 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
W20260927 21:33:25.007798 7338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47886
W20260927 21:33:25.008690 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:33:25.008800 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7215
I20260927 21:33:25.022835 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 6855
I20260927 21:33:25.023648 6917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:25.120481 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 6855
2026-09-27T21:33:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3363 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:25 dist-test-slave-pqr8 krb5kdc[7360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:25 dist-test-slave-pqr8 krb5kdc[7360](info): set up 2 sockets
Sep 27 21:33:25 dist-test-slave-pqr8 krb5kdc[7360](info): commencing operation
krb5kdc: starting...
W20260927 21:33:26.181506 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:26 dist-test-slave-pqr8 krb5kdc[7360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:26.352746 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44009
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:26.648582 7373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:26.648881 7373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:26.649044 7373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:26.655874 7373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:26.656005 7373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:26.656105 7373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:26.656205 7373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:26.656289 7373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:26.665079 7373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:26.666281 7373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:26.667490 7373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:26.678350 7378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:26.678297 7379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:26.678375 7381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:26.679350 7373 server_base.cc:1034] running on GCE node
I20260927 21:33:26.680109 7373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:26.681281 7373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:26.682489 7373 hybrid_clock.cc:648] HybridClock initialized: now 1790544806682425 us; error 54 us; skew 500 ppm
I20260927 21:33:26.682807 7373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:26 dist-test-slave-pqr8 krb5kdc[7360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:26.691335 7373 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:26.692672 7373 webserver.cc:460] Webserver started at http://127.3.186.254:41137/ using document root <none> and password file <none>
I20260927 21:33:26.693219 7373 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:26.693312 7373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:26.693622 7373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:26.696272 7373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "44cd661c0b4148feaf598fd67961667f"
format_stamp: "Formatted at 2026-09-27 21:33:26 on dist-test-slave-pqr8"
server_key: "0d16115467e9ba22281ab21887a08633"
server_key_iv: "3f1bad7dabaa69570763ab267f5a3d1a"
server_key_version: "encryptionkey@0"
I20260927 21:33:26.696952 7373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "44cd661c0b4148feaf598fd67961667f"
format_stamp: "Formatted at 2026-09-27 21:33:26 on dist-test-slave-pqr8"
server_key: "0d16115467e9ba22281ab21887a08633"
server_key_iv: "3f1bad7dabaa69570763ab267f5a3d1a"
server_key_version: "encryptionkey@0"
I20260927 21:33:26.701381 7373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 21:33:26.704389 7388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:26.705577 7373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:26.705732 7373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "44cd661c0b4148feaf598fd67961667f"
format_stamp: "Formatted at 2026-09-27 21:33:26 on dist-test-slave-pqr8"
server_key: "0d16115467e9ba22281ab21887a08633"
server_key_iv: "3f1bad7dabaa69570763ab267f5a3d1a"
server_key_version: "encryptionkey@0"
I20260927 21:33:26.705901 7373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:26.737929 7373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:26.741641 7373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:26.741916 7373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:26.759841 7373 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44009
I20260927 21:33:26.759909 7439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44009 every 8 connection(s)
I20260927 21:33:26.760991 7373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:26.764189 7440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:26.766530 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7373
I20260927 21:33:26.766742 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:26.767479 3819 external_mini_cluster.cc:1428] Setting key 273c3b7e4dc3900802309832ad8aac19
I20260927 21:33:26.775956 7440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f: Bootstrap starting.
I20260927 21:33:26.779374 7440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:26.780627 7440 log.cc:824] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:33:26 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:26.783159 7440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f: No bootstrap required, opened a new log
I20260927 21:33:26.788820 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:26.769411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.3.186.254:44009)
0927 21:33:26.770193 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:26.770203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:26.770241 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:33:26.771753 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:26.771771 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:26.772090 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:26.772379 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:26.773950 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:26.775163 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:26.776858 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:26.777491 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:26.777668 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:26.782191 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:26.782197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:26.782209 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:26.782238 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:26.785119 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:26.785844 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:26.785850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:26.785874 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:26.786027 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:26.786402 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:26.786411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:26.786413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:26.786884 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:26.787183 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:26.787621 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 21:33:26.787954 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":362,"threads_started":1}
I20260927 21:33:26.790789 7440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44cd661c0b4148feaf598fd67961667f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44009 } }
I20260927 21:33:26.791111 7440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:26.791224 7440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cd661c0b4148feaf598fd67961667f, State: Initialized, Role: FOLLOWER
I20260927 21:33:26.791746 7440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44cd661c0b4148feaf598fd67961667f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44009 } }
I20260927 21:33:26.791934 7440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:26.792091 7440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:26.792245 7440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:26.794175 7440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44cd661c0b4148feaf598fd67961667f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44009 } }
I20260927 21:33:26.794617 7440 leader_election.cc:302] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44cd661c0b4148feaf598fd67961667f; no voters:
I20260927 21:33:26.795195 7440 leader_election.cc:288] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:26.795279 7445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:26.796947 7445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [term 1 LEADER]: Becoming Leader. State: Replica: 44cd661c0b4148feaf598fd67961667f, State: Running, Role: LEADER
I20260927 21:33:26.797129 7440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:26.797416 7445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44cd661c0b4148feaf598fd67961667f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44009 } }
I20260927 21:33:26.800354 7446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44cd661c0b4148feaf598fd67961667f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44cd661c0b4148feaf598fd67961667f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44009 } } }
I20260927 21:33:26.800335 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44cd661c0b4148feaf598fd67961667f. Latest consensus state: current_term: 1 leader_uuid: "44cd661c0b4148feaf598fd67961667f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44cd661c0b4148feaf598fd67961667f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44009 } } }
I20260927 21:33:26.800756 7446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:26.800765 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:26.806588 7454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:26.810923 7454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:26.818491 7454 catalog_manager.cc:1347] Generated new cluster ID: 0f44b01e14ff4dc2b6c64684b9ca269b
I20260927 21:33:26.818621 7454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:26.851881 7454 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:26.852991 7454 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:26.862637 7454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44cd661c0b4148feaf598fd67961667f: Generated new TSK 0
I20260927 21:33:26.863314 7454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:26.914916 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44009
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:27.227712 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:27.227994 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:27.228163 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:27.234819 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:27.234943 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:27.235122 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:27.244086 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:27.245157 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:27.246361 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:27.258731 7474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:27.258693 7472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:27.258769 7471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:27.259528 7466 server_base.cc:1034] running on GCE node
I20260927 21:33:27.259987 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:27.260820 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:27.262009 7466 hybrid_clock.cc:648] HybridClock initialized: now 1790544807261954 us; error 50 us; skew 500 ppm
I20260927 21:33:27.262287 7466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:27 dist-test-slave-pqr8 krb5kdc[7360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:27.265125 7466 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:27.266391 7466 webserver.cc:460] Webserver started at http://127.3.186.193:40051/ using document root <none> and password file <none>
I20260927 21:33:27.267002 7466 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:27.267143 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:27.267477 7466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:27.270114 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "2effe3480422481d90417df49b5d066c"
format_stamp: "Formatted at 2026-09-27 21:33:27 on dist-test-slave-pqr8"
server_key: "46311097632f2c52dbd480b6ec2eb74b"
server_key_iv: "5ce62dd83a8c2c1283f4c4ec659a6525"
server_key_version: "encryptionkey@0"
I20260927 21:33:27.270855 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "2effe3480422481d90417df49b5d066c"
format_stamp: "Formatted at 2026-09-27 21:33:27 on dist-test-slave-pqr8"
server_key: "46311097632f2c52dbd480b6ec2eb74b"
server_key_iv: "5ce62dd83a8c2c1283f4c4ec659a6525"
server_key_version: "encryptionkey@0"
I20260927 21:33:27.275341 7466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 21:33:27.278385 7481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:27.279469 7466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:27.279631 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "2effe3480422481d90417df49b5d066c"
format_stamp: "Formatted at 2026-09-27 21:33:27 on dist-test-slave-pqr8"
server_key: "46311097632f2c52dbd480b6ec2eb74b"
server_key_iv: "5ce62dd83a8c2c1283f4c4ec659a6525"
server_key_version: "encryptionkey@0"
I20260927 21:33:27.279814 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:27.301818 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:27.305874 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:27.306154 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:27.307050 7466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:27.308244 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:27.308346 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:27.308490 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:27.308562 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:27.339813 7466 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:36457
I20260927 21:33:27.339892 7593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:36457 every 8 connection(s)
I20260927 21:33:27.340987 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:27.350755 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7466
I20260927 21:33:27.350965 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:27.351583 3819 external_mini_cluster.cc:1428] Setting key 6c1b3abd49050678f1feaa9cc6049d61
Sep 27 21:33:27 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:27.360258 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:27.343415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:60219 (local address 127.3.186.254:44009)
0927 21:33:27.343896 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:27.343901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:27.344824 (+ 923us) server_negotiation.cc:407] Connection header received
0927 21:33:27.346616 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:27.346622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:27.346709 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:27.346824 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:27.348200 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.348913 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.350429 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.350834 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.350975 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:27.354775 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:27.354780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:27.354783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:27.354810 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:27.357273 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:27.358154 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:27.358157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:27.358159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:27.358260 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:27.358672 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:27.358679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:27.358683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:27.359022 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:27.359175 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:27.359736 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 21:33:27.359925 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":277,"threads_started":1}
I20260927 21:33:27.361120 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:27.343917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44009 (local address 127.3.186.193:60219)
0927 21:33:27.344663 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:27.344698 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:27.346356 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:27.347039 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:27.347048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:27.347538 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:27.348048 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:27.348064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.349056 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.349062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:27.350276 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:27.350287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.351045 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.351052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:27.351277 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:27.351881 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:27.351900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:27.354569 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:27.357403 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:27.357413 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:27.357429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:27.358039 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:27.358398 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:27.358402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:27.358404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:27.358559 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:27.359151 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:27.359156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:27.359404 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 21:33:27.359646 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 21:33:27.360093 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":265,"threads_started":1}
I20260927 21:33:27.362612 7594 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44009
I20260927 21:33:27.362892 7594 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:27.363610 7594 heartbeater.cc:511] Master 127.3.186.254:44009 requested a full tablet report, sending...
I20260927 21:33:27.365813 7405 ts_manager.cc:194] Registered new tserver with Master: 2effe3480422481d90417df49b5d066c (127.3.186.193:36457)
I20260927 21:33:27.367925 7405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:60219
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:33:27.392725 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44009
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:27.687496 7600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:27.687767 7600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:27.687939 7600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:27.694617 7600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:27.694754 7600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:27.694921 7600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:27.703688 7600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:27.704711 7600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:27.705967 7600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:27.719924 7605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:27.720813 7606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:27.721632 7608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:27.722692 7600 server_base.cc:1034] running on GCE node
I20260927 21:33:27.723155 7600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:27.723888 7600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:27.725104 7600 hybrid_clock.cc:648] HybridClock initialized: now 1790544807725064 us; error 53 us; skew 500 ppm
I20260927 21:33:27.725386 7600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:27 dist-test-slave-pqr8 krb5kdc[7360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:27.728293 7600 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:27.729615 7600 webserver.cc:460] Webserver started at http://127.3.186.194:43939/ using document root <none> and password file <none>
I20260927 21:33:27.730243 7600 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:27.730370 7600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:27.730629 7600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:27.733011 7600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "ed70532081fb4dde834ab40b839f6a29"
format_stamp: "Formatted at 2026-09-27 21:33:27 on dist-test-slave-pqr8"
server_key: "ea000ba7633388632c2cc43c198c5a7f"
server_key_iv: "1c355d59f485aecb57f5e0e2dd87d46d"
server_key_version: "encryptionkey@0"
I20260927 21:33:27.733752 7600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "ed70532081fb4dde834ab40b839f6a29"
format_stamp: "Formatted at 2026-09-27 21:33:27 on dist-test-slave-pqr8"
server_key: "ea000ba7633388632c2cc43c198c5a7f"
server_key_iv: "1c355d59f485aecb57f5e0e2dd87d46d"
server_key_version: "encryptionkey@0"
I20260927 21:33:27.738121 7600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:33:27.741093 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:27.742270 7600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 21:33:27.742451 7600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "ed70532081fb4dde834ab40b839f6a29"
format_stamp: "Formatted at 2026-09-27 21:33:27 on dist-test-slave-pqr8"
server_key: "ea000ba7633388632c2cc43c198c5a7f"
server_key_iv: "1c355d59f485aecb57f5e0e2dd87d46d"
server_key_version: "encryptionkey@0"
I20260927 21:33:27.742620 7600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:27.764096 7600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:27.767755 7600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:27.768008 7600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:27.768867 7600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:27.770011 7600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:27.770108 7600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:27.770241 7600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:27.770309 7600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:27.803990 7600 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:36909
I20260927 21:33:27.804065 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:36909 every 8 connection(s)
I20260927 21:33:27.805220 7600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:27.807350 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7600
I20260927 21:33:27.807513 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:27.807963 3819 external_mini_cluster.cc:1428] Setting key c02a218d4919a2490606ee1633a67055
Sep 27 21:33:27 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544807, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:27.824208 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:27.807685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:57855 (local address 127.3.186.254:44009)
0927 21:33:27.807864 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:27.807871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:27.809022 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 21:33:27.810810 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:27.810815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:27.810913 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:27.811032 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:27.812419 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.813132 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.815055 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.815445 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.815542 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:27.819060 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:27.819065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:27.819068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:27.819094 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:27.821456 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:27.822140 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:27.822145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:27.822149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:27.822268 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:27.822665 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:27.822670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:27.822673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:27.822904 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:27.823057 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:27.823682 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 21:33:27.823884 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:33:27.825065 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:27.808158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44009 (local address 127.3.186.194:57855)
0927 21:33:27.808875 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:27.808920 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:27.810567 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:27.811253 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:27.811261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:27.811724 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:27.812253 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:27.812272 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.813268 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.813273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:27.814867 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:27.814884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:27.815629 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:27.815634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:27.815862 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:27.816664 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:27.816691 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:27.818852 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:27.821637 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:27.821645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:27.821659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:27.822021 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:27.822373 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:27.822378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:27.822382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:27.822548 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:27.823023 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:27.823030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:27.823320 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 21:33:27.823570 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 21:33:27.824004 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
I20260927 21:33:27.826375 7728 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44009
I20260927 21:33:27.826647 7728 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:27.827323 7728 heartbeater.cc:511] Master 127.3.186.254:44009 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
I20260927 21:33:27.828791 7405 ts_manager.cc:194] Registered new tserver with Master: ed70532081fb4dde834ab40b839f6a29 (127.3.186.194:36909)
I20260927 21:33:27.830008 7405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:57855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:33:27.848053 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44009
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:28.143433 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:28.143672 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:28.143839 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:28.151209 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:28.151295 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:28.151423 7733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:28.160205 7733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:28.161226 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:28.162410 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:28.174778 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:28.174736 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:28.174815 7741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:28.175582 7733 server_base.cc:1034] running on GCE node
I20260927 21:33:28.176195 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:28.176981 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:28.178181 7733 hybrid_clock.cc:648] HybridClock initialized: now 1790544808178146 us; error 34 us; skew 500 ppm
I20260927 21:33:28.178471 7733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:28.181389 7733 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:28.182746 7733 webserver.cc:460] Webserver started at http://127.3.186.195:41741/ using document root <none> and password file <none>
I20260927 21:33:28.183394 7733 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:28.183487 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:28.183715 7733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:28.186429 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "72327f1316b14990a256371b5317b204"
format_stamp: "Formatted at 2026-09-27 21:33:28 on dist-test-slave-pqr8"
server_key: "1c55d2fc03e97193201f6880a798e562"
server_key_iv: "54be93cd809bee85040933cd0d014a9a"
server_key_version: "encryptionkey@0"
I20260927 21:33:28.187096 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "72327f1316b14990a256371b5317b204"
format_stamp: "Formatted at 2026-09-27 21:33:28 on dist-test-slave-pqr8"
server_key: "1c55d2fc03e97193201f6880a798e562"
server_key_iv: "54be93cd809bee85040933cd0d014a9a"
server_key_version: "encryptionkey@0"
I20260927 21:33:28.191967 7733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:33:28.195004 7748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:28.196192 7733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:28.196360 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "72327f1316b14990a256371b5317b204"
format_stamp: "Formatted at 2026-09-27 21:33:28 on dist-test-slave-pqr8"
server_key: "1c55d2fc03e97193201f6880a798e562"
server_key_iv: "54be93cd809bee85040933cd0d014a9a"
server_key_version: "encryptionkey@0"
I20260927 21:33:28.196549 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:28.215265 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:28.218734 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:28.218997 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:28.219962 7733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:28.221468 7733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:28.221604 7733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:28.221773 7733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:28.221848 7733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:28.253715 7733 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:33519
I20260927 21:33:28.253752 7860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:33519 every 8 connection(s)
I20260927 21:33:28.254844 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:28.261256 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7733
I20260927 21:33:28.261416 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:28.261819 3819 external_mini_cluster.cc:1428] Setting key 367ff8d629c35bb90a3542aa8db2cf48
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:28.272136 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.257066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:52489 (local address 127.3.186.254:44009)
0927 21:33:28.257237 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:28.257242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:28.258306 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 21:33:28.259945 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:28.259950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:28.260031 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:28.260147 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:28.261485 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.262244 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.263629 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.264040 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.264169 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.267284 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:28.267289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:28.267293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:28.267334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:28.269481 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.270151 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.270158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.270164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.270280 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.270657 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.270663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.270666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.270911 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:28.271065 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:28.271611 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 21:33:28.271814 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:33:28.272974 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.257496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44009 (local address 127.3.186.195:52489)
0927 21:33:28.258160 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:28.258195 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:28.259705 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:28.260361 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:28.260368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:28.260828 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:28.261298 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:28.261344 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.262379 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.262383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:28.263489 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:28.263499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.264248 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.264257 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:28.264476 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.265032 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:28.265056 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:28.267081 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:28.269643 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:28.269650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:28.269663 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:28.270035 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:28.270384 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:28.270389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:28.270393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:28.270552 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:28.271029 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:28.271037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:28.271314 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 21:33:28.271549 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 21:33:28.271975 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":234,"threads_started":1}
I20260927 21:33:28.274181 7861 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44009
I20260927 21:33:28.274487 7861 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:28.275296 7861 heartbeater.cc:511] Master 127.3.186.254:44009 requested a full tablet report, sending...
I20260927 21:33:28.276723 7405 ts_manager.cc:194] Registered new tserver with Master: 72327f1316b14990a256371b5317b204 (127.3.186.195:33519)
I20260927 21:33:28.277474 7405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:52489
I20260927 21:33:28.287946 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:28.331713 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.318649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.3.186.193:36457)
0927 21:33:28.319226 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:28.319232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:28.319338 (+ 106us) server_negotiation.cc:407] Connection header received
0927 21:33:28.319468 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:28.319474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:28.319656 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:28.319768 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:28.320950 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.321946 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.323051 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.323508 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.323595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.326953 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:28.326962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:28.326973 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:28.327007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:28.329376 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.330002 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.330008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.330011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.330117 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.330440 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.330443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.330445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.330731 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:28.330948 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:28.331147 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 21:33:28.331310 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":286,"threads_started":1}
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:28.348865 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.335357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.3.186.254:44009)
0927 21:33:28.335549 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:28.335555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:28.335743 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:33:28.335922 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:28.335927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:28.335995 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:28.336095 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:28.337216 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.337963 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.339098 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.339566 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.339677 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.343398 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:28.343405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:28.343408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:28.343448 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:28.346627 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.347285 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.347290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.347293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.347429 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.347736 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.347741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.347744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.348098 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:28.348321 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:28.348518 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 21:33:28.348649 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 21:33:28.349344 7405 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49534
I20260927 21:33:28.350185 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7466
I20260927 21:33:28.361140 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.350960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.3.186.193:36457)
0927 21:33:28.351126 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:28.351130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:28.351313 (+ 183us) server_negotiation.cc:407] Connection header received
0927 21:33:28.351470 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:28.351480 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:28.351548 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:28.351667 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:28.352773 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.353406 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.354483 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.354841 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.354959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.356868 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:28.356872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:28.356875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:28.356900 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:28.359018 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.359573 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.359579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.359582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.359683 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.360008 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.360013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.360016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.360227 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:28.360440 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:28.360710 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 21:33:28.360900 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:33:28.361662 7589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:28.439438 7594 heartbeater.cc:503] Master 127.3.186.254:44009 was elected leader, sending a full tablet report...
I20260927 21:33:28.439921 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7466
W20260927 21:33:28.455143 7389 connection.cc:535] server connection from 127.3.186.193:60219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:33:28.455456 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7600
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:28.470034 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.456254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.3.186.194:36909)
0927 21:33:28.456731 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:28.456739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:28.456764 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:33:28.456817 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:28.456823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:28.457020 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:28.457139 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:28.458259 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.459207 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.460456 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.460979 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.461115 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.464847 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:28.464857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:28.464869 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:28.464905 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:28.467459 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.468044 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.468051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.468053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.468160 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.468531 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.468537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.468540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.468843 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:28.469126 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:28.469449 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:33:28.469723 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":235,"threads_started":1}
I20260927 21:33:28.471168 7723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:28.548360 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7600
I20260927 21:33:28.564342 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7733
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:28.579046 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:28.565306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.3.186.195:33519)
0927 21:33:28.565924 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:28.565931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:28.565958 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:33:28.566013 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:28.566019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:28.566227 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:28.566363 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:28.567501 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.568452 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.569711 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:28.570224 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:28.570321 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:28.574165 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:28.574174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:28.574186 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:28.574217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:28.576906 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.577459 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.577465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.577467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.577633 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:28.577903 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:28.577905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:28.577907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:28.578206 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:28.578417 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:28.578627 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 21:33:28.578794 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":342,"threads_started":1}
I20260927 21:33:28.580190 7856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:28.658712 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7733
I20260927 21:33:28.673699 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7373
I20260927 21:33:28.674747 7435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:28.770219 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7373
2026-09-27T21:33:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3649 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7880](info): set up 2 sockets
Sep 27 21:33:28 dist-test-slave-pqr8 krb5kdc[7880](info): commencing operation
krb5kdc: starting...
W20260927 21:33:29.825735 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:29 dist-test-slave-pqr8 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:29.994407 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:45313
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:30.292601 7893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:30.292833 7893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:30.292985 7893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:30.299690 7893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:30.299772 7893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:30.299849 7893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:30.299955 7893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:30.300016 7893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:30.308992 7893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:30.310176 7893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:30.311398 7893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:30.321831 7898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:30.321923 7899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:30.322448 7901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:30.322728 7893 server_base.cc:1034] running on GCE node
I20260927 21:33:30.323382 7893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:30.324545 7893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:30.325757 7893 hybrid_clock.cc:648] HybridClock initialized: now 1790544810325722 us; error 50 us; skew 500 ppm
I20260927 21:33:30.326067 7893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:30 dist-test-slave-pqr8 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544810, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:30.328970 7893 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:30.330034 7893 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:30.330156 7893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:30.330506 7893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:30.333099 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "c1dfb65e25c6443bac1b62dcb75c3321"
format_stamp: "Formatted at 2026-09-27 21:33:30 on dist-test-slave-pqr8"
server_key: "d0accb2c6c5cfbf355296fd993f4512e"
server_key_iv: "f6537fb7002024b5154c44bb60b3abda"
server_key_version: "encryptionkey@0"
I20260927 21:33:30.333884 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "c1dfb65e25c6443bac1b62dcb75c3321"
format_stamp: "Formatted at 2026-09-27 21:33:30 on dist-test-slave-pqr8"
server_key: "d0accb2c6c5cfbf355296fd993f4512e"
server_key_iv: "f6537fb7002024b5154c44bb60b3abda"
server_key_version: "encryptionkey@0"
I20260927 21:33:30.338444 7893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:30.341470 7907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:30.342651 7893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:30.342821 7893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "c1dfb65e25c6443bac1b62dcb75c3321"
format_stamp: "Formatted at 2026-09-27 21:33:30 on dist-test-slave-pqr8"
server_key: "d0accb2c6c5cfbf355296fd993f4512e"
server_key_iv: "f6537fb7002024b5154c44bb60b3abda"
server_key_version: "encryptionkey@0"
I20260927 21:33:30.343034 7893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:30.369063 7893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:30.373044 7893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:30.373294 7893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:30.390103 7893 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:45313
I20260927 21:33:30.390110 7958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:45313 every 8 connection(s)
I20260927 21:33:30.390869 7893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:30.394063 7959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:30.398639 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7893
I20260927 21:33:30.398866 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:30.399621 3819 external_mini_cluster.cc:1428] Setting key fa86e1064676d1d97f0345f3b9de7b04
I20260927 21:33:30.404696 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321: Bootstrap starting.
I20260927 21:33:30.408150 7959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:30.409260 7959 log.cc:824] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:30.411891 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321: No bootstrap required, opened a new log
Sep 27 21:33:30 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:30.421794 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:30.401591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.3.186.254:45313)
0927 21:33:30.402332 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:30.402342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:30.402383 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:33:30.403953 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:30.403975 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:30.404302 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:30.404608 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:30.406357 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:30.407641 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:30.409446 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:30.410134 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:30.410263 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:30.413928 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:30.413937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:30.413954 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:30.414009 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:30.417823 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:30.418417 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:30.418425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:30.418432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:30.418630 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:30.418955 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:30.418960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:30.418963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:30.419614 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:30.419886 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:30.420169 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 21:33:30.420567 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":355,"threads_started":1}
I20260927 21:33:30.421733 7959 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45313 } }
I20260927 21:33:30.422137 7959 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:30.422250 7959 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1dfb65e25c6443bac1b62dcb75c3321, State: Initialized, Role: FOLLOWER
I20260927 21:33:30.422827 7959 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45313 } }
I20260927 21:33:30.423097 7959 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:30.423246 7959 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:30.423439 7959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:30.425792 7959 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45313 } }
I20260927 21:33:30.426245 7959 leader_election.cc:302] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1dfb65e25c6443bac1b62dcb75c3321; no voters:
I20260927 21:33:30.426812 7959 leader_election.cc:288] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:30.426901 7964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:30.427700 7964 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [term 1 LEADER]: Becoming Leader. State: Replica: c1dfb65e25c6443bac1b62dcb75c3321, State: Running, Role: LEADER
I20260927 21:33:30.428280 7964 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45313 } }
I20260927 21:33:30.429005 7959 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:30.431209 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45313 } } }
I20260927 21:33:30.431547 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:30.431305 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1dfb65e25c6443bac1b62dcb75c3321. Latest consensus state: current_term: 1 leader_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dfb65e25c6443bac1b62dcb75c3321" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45313 } } }
I20260927 21:33:30.431716 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:30.434521 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:30.439013 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:30.447573 7971 catalog_manager.cc:1347] Generated new cluster ID: 02aa45efb41e4f31ba30b72b0cb01e96
I20260927 21:33:30.447665 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:30.465442 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:30.467103 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:30.479566 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321: Generated new TSK 0
I20260927 21:33:30.480425 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:30.521271 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:45313
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:30.823204 7985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:30.823433 7985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:30.823519 7985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:30.830540 7985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:30.830626 7985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:30.830741 7985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:30.839604 7985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:30.840641 7985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:30.842044 7985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:30.854833 7990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:30.854866 7993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:30.854935 7991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:30.855464 7985 server_base.cc:1034] running on GCE node
I20260927 21:33:30.856007 7985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:30.856823 7985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:30.858028 7985 hybrid_clock.cc:648] HybridClock initialized: now 1790544810857984 us; error 58 us; skew 500 ppm
I20260927 21:33:30.858322 7985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:30 dist-test-slave-pqr8 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544810, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:30.861090 7985 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:30.862123 7985 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:30.862246 7985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:30.862534 7985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:30.864974 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "0cdfa1cbb0d54a6881022cc0e7c66674"
format_stamp: "Formatted at 2026-09-27 21:33:30 on dist-test-slave-pqr8"
server_key: "041adced06088c2b229491c3401dd998"
server_key_iv: "8943aa8371b23ca227546cb7cdb423d1"
server_key_version: "encryptionkey@0"
I20260927 21:33:30.865792 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "0cdfa1cbb0d54a6881022cc0e7c66674"
format_stamp: "Formatted at 2026-09-27 21:33:30 on dist-test-slave-pqr8"
server_key: "041adced06088c2b229491c3401dd998"
server_key_iv: "8943aa8371b23ca227546cb7cdb423d1"
server_key_version: "encryptionkey@0"
I20260927 21:33:30.870100 7985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:30.873112 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:30.874360 7985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:33:30.874543 7985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "0cdfa1cbb0d54a6881022cc0e7c66674"
format_stamp: "Formatted at 2026-09-27 21:33:30 on dist-test-slave-pqr8"
server_key: "041adced06088c2b229491c3401dd998"
server_key_iv: "8943aa8371b23ca227546cb7cdb423d1"
server_key_version: "encryptionkey@0"
I20260927 21:33:30.874754 7985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:30.897684 7985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:30.901219 7985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:30.901469 7985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:30.902287 7985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:30.903465 7985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:30.903549 7985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:30.903621 7985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:30.903674 7985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:30.935130 7985 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:39219
I20260927 21:33:30.935217 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:39219 every 8 connection(s)
I20260927 21:33:30.935909 7985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:30.946123 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 7985
I20260927 21:33:30.946344 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:30.947149 3819 external_mini_cluster.cc:1428] Setting key 2e30f6c72c22a60108bebbe96a37f3b2
Sep 27 21:33:30 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544810, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:30.953868 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:30.938103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:33585 (local address 127.3.186.254:45313)
0927 21:33:30.938489 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:30.938495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:30.939351 (+ 856us) server_negotiation.cc:407] Connection header received
0927 21:33:30.940934 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:30.940938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:30.941015 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:30.941107 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:30.942444 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:30.943129 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:30.944496 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:30.944870 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:30.944958 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:30.948526 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:30.948530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:30.948533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:30.948561 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:30.951102 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:30.951790 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:30.951797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:30.951800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:30.951924 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:30.952312 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:30.952318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:30.952321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:30.952562 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:30.952711 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:30.953289 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 21:33:30.953504 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":185,"threads_started":1}
I20260927 21:33:30.954660 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:30.938599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:45313 (local address 127.3.186.193:33585)
0927 21:33:30.939203 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:30.939237 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:30.940683 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:30.941283 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:30.941291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:30.941830 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:30.942313 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:30.942329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:30.943249 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:30.943252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:30.944386 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:30.944396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:30.945071 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:30.945075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:30.945243 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:30.945864 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:30.945895 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:30.948322 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:30.951265 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:30.951273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:30.951286 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:30.951671 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:30.952034 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:30.952038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:30.952042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:30.952202 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:30.952692 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:30.952699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:30.952981 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 21:33:30.953217 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 21:33:30.953704 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":250,"threads_started":1}
I20260927 21:33:30.956017 8112 heartbeater.cc:347] Connected to a master server at 127.3.186.254:45313
I20260927 21:33:30.956300 8112 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:30.956876 8112 heartbeater.cc:511] Master 127.3.186.254:45313 requested a full tablet report, sending...
I20260927 21:33:30.958989 7924 ts_manager.cc:194] Registered new tserver with Master: 0cdfa1cbb0d54a6881022cc0e7c66674 (127.3.186.193:39219)
I20260927 21:33:30.960397 7924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:33585
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:33:30.986768 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:45313
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:31.284835 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:31.285084 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:31.285171 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:31.291817 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:31.291896 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:31.291994 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:31.300514 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:31.301455 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:31.302572 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:31.314872 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:31.314913 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:31.314934 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:31.315570 8118 server_base.cc:1034] running on GCE node
I20260927 21:33:31.316085 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:31.316905 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:31.318112 8118 hybrid_clock.cc:648] HybridClock initialized: now 1790544811318064 us; error 61 us; skew 500 ppm
I20260927 21:33:31.318486 8118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:31 dist-test-slave-pqr8 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:31.321238 8118 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:31.322327 8118 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:31.322443 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:31.322739 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:31.325213 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "8744efe6708b4e7397f8ddc909a77943"
format_stamp: "Formatted at 2026-09-27 21:33:31 on dist-test-slave-pqr8"
server_key: "960ffbbe4038c781015f9917e35ff15a"
server_key_iv: "b93a95f6619e74eeb0ac5e382e76e027"
server_key_version: "encryptionkey@0"
I20260927 21:33:31.325973 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "8744efe6708b4e7397f8ddc909a77943"
format_stamp: "Formatted at 2026-09-27 21:33:31 on dist-test-slave-pqr8"
server_key: "960ffbbe4038c781015f9917e35ff15a"
server_key_iv: "b93a95f6619e74eeb0ac5e382e76e027"
server_key_version: "encryptionkey@0"
I20260927 21:33:31.330492 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 21:33:31.333498 8132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:31.334761 8118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:31.334915 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "8744efe6708b4e7397f8ddc909a77943"
format_stamp: "Formatted at 2026-09-27 21:33:31 on dist-test-slave-pqr8"
server_key: "960ffbbe4038c781015f9917e35ff15a"
server_key_iv: "b93a95f6619e74eeb0ac5e382e76e027"
server_key_version: "encryptionkey@0"
I20260927 21:33:31.335112 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:31.365291 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:31.368625 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:31.368866 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:31.369702 8118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:31.370848 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:31.370956 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:31.371071 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:31.371160 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:31.405203 8118 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:41601
I20260927 21:33:31.405269 8244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:41601 every 8 connection(s)
I20260927 21:33:31.405995 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:31.410357 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8118
I20260927 21:33:31.410537 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:31.411062 3819 external_mini_cluster.cc:1428] Setting key bc25d1946a12edab2b75b33dc975db70
Sep 27 21:33:31 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:31.426865 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.408206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:45921 (local address 127.3.186.254:45313)
0927 21:33:31.408384 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:31.408390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:31.409420 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 21:33:31.411625 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:31.411631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:31.411717 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:31.411831 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:31.413472 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.414212 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.416488 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.416898 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.416995 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.420497 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:31.420506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:31.420514 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:31.420553 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:31.423960 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:31.424591 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:31.424599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:31.424612 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:31.424750 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:31.425112 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:31.425119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:31.425125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:31.425472 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:31.425663 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:31.426194 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 21:33:31.426427 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:33:31.427412 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.408617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:45313 (local address 127.3.186.194:45921)
0927 21:33:31.409273 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:31.409312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:31.411315 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:31.412079 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:31.412089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:31.412611 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:31.413302 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.413326 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.414361 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.414369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.416340 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.416356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.417112 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.417121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.417417 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.418088 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:31.418120 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:31.420302 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:31.424082 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:31.424088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:31.424097 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:31.424477 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:31.424849 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:31.424852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:31.424854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:31.424993 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:31.425619 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:31.425624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:31.425871 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 21:33:31.426074 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 21:33:31.426378 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":253,"threads_started":1}
I20260927 21:33:31.428663 8245 heartbeater.cc:347] Connected to a master server at 127.3.186.254:45313
I20260927 21:33:31.428956 8245 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:31.429555 8245 heartbeater.cc:511] Master 127.3.186.254:45313 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:33:31.430903 7924 ts_manager.cc:194] Registered new tserver with Master: 8744efe6708b4e7397f8ddc909a77943 (127.3.186.194:41601)
I20260927 21:33:31.431794 7924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:45921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:33:31.452589 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:45313
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:31.747565 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:31.747784 8250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:31.747949 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:31.754652 8250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:31.754756 8250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:31.754925 8250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:31.763397 8250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:31.764417 8250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:31.765743 8250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:31.777467 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:31.777405 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:31.777443 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:31.778605 8250 server_base.cc:1034] running on GCE node
I20260927 21:33:31.779238 8250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:31.780087 8250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:31.781279 8250 hybrid_clock.cc:648] HybridClock initialized: now 1790544811781246 us; error 32 us; skew 500 ppm
I20260927 21:33:31.781642 8250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:31 dist-test-slave-pqr8 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:31.790548 8250 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:31.791620 8250 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:31.791742 8250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:31.792043 8250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:31.794833 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "d93e19841b244ee9b037724007eb5798"
format_stamp: "Formatted at 2026-09-27 21:33:31 on dist-test-slave-pqr8"
server_key: "d1ffbe0ce004d38aa2932f1aba2aeb8c"
server_key_iv: "8442cdf558a5c9ad0be9d0e4a708e32b"
server_key_version: "encryptionkey@0"
I20260927 21:33:31.795643 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "d93e19841b244ee9b037724007eb5798"
format_stamp: "Formatted at 2026-09-27 21:33:31 on dist-test-slave-pqr8"
server_key: "d1ffbe0ce004d38aa2932f1aba2aeb8c"
server_key_iv: "8442cdf558a5c9ad0be9d0e4a708e32b"
server_key_version: "encryptionkey@0"
I20260927 21:33:31.800292 8250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:33:31.803428 8264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:31.804629 8250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:31.804788 8250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "d93e19841b244ee9b037724007eb5798"
format_stamp: "Formatted at 2026-09-27 21:33:31 on dist-test-slave-pqr8"
server_key: "d1ffbe0ce004d38aa2932f1aba2aeb8c"
server_key_iv: "8442cdf558a5c9ad0be9d0e4a708e32b"
server_key_version: "encryptionkey@0"
I20260927 21:33:31.804996 8250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:31.820724 8250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:31.824157 8250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:31.824409 8250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:31.825232 8250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:31.826417 8250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:31.826536 8250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:31.826669 8250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:31.826754 8250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:31.857710 8250 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:44809
I20260927 21:33:31.857779 8376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:44809 every 8 connection(s)
I20260927 21:33:31.858471 8250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:31.866304 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8250
I20260927 21:33:31.866495 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:31.866998 3819 external_mini_cluster.cc:1428] Setting key fbd59426ca2ef9a088b905309000c1a6
Sep 27 21:33:31 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544811, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:31.876744 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.860559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:48819 (local address 127.3.186.254:45313)
0927 21:33:31.860751 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:31.860757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:31.861812 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 21:33:31.863358 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:31.863362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:31.863422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:31.863503 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:31.864733 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.865411 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.867557 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.868169 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.868332 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.871704 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:31.871709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:31.871712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:31.871737 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:31.874120 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:31.874774 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:31.874778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:31.874781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:31.874883 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:31.875258 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:31.875263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:31.875266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:31.875491 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:31.875629 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:31.876203 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 21:33:31.876391 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:33:31.877563 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.860967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:45313 (local address 127.3.186.195:48819)
0927 21:33:31.861676 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:31.861714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:31.863140 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:31.863675 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:31.863682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:31.864129 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:31.864600 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.864619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.865589 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.865595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.867412 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.867426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.868428 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.868436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.868707 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.869319 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:31.869348 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:31.871501 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:31.874262 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:31.874270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:31.874282 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:31.874645 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:31.874978 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:31.874983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:31.874986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:31.875152 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:31.875607 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:31.875613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:31.875893 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 21:33:31.876117 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 21:33:31.876539 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260927 21:33:31.878714 8377 heartbeater.cc:347] Connected to a master server at 127.3.186.254:45313
I20260927 21:33:31.878927 8377 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:31.879479 8377 heartbeater.cc:511] Master 127.3.186.254:45313 requested a full tablet report, sending...
I20260927 21:33:31.880645 7924 ts_manager.cc:194] Registered new tserver with Master: d93e19841b244ee9b037724007eb5798 (127.3.186.195:44809)
I20260927 21:33:31.881570 7924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:48819
I20260927 21:33:31.882180 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:33:31.898026 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.886938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.3.186.254:45313)
0927 21:33:31.887074 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:31.887078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:31.887556 (+ 478us) server_negotiation.cc:407] Connection header received
0927 21:33:31.887730 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:31.887736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:31.887803 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:31.887907 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:31.889226 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.889951 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.891128 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.891521 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.891640 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.893488 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:31.893494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:31.893498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:31.893601 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:31.895611 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:31.896209 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:31.896216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:31.896219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:31.896340 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:31.896692 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:31.896699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:31.896704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:31.896965 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:31.897190 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:31.897494 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:33:31.897732 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:33:31.901343 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:31.908036 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:31.927953 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.918406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:41601 (local address 127.0.0.1:56934)
0927 21:33:31.919169 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:31.919216 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:31.919657 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:31.920468 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:31.920476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:31.920513 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:31.920845 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.920856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.923011 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.923018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.925328 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.925341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.927102 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.927109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.927302 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.927345 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:33:31.927477 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:33:31.927643 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":402,"threads_started":1}
I20260927 21:33:31.928169 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.918732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.3.186.194:41601)
0927 21:33:31.919287 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:31.919295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:31.919323 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:33:31.919977 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:31.919986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:31.920246 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:31.921025 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:31.921287 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.922839 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.925483 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.926897 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.927028 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.927228 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:31.927543 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 21:33:31.927709 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":951,"threads_started":1}
I20260927 21:33:31.928424 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.919492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:44809 (local address 127.0.0.1:41542)
0927 21:33:31.920222 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:31.920238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:31.920357 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:31.921446 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:31.921452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:31.921476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:31.921937 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.921949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.923965 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.923974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.925885 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.925898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.927643 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.927650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.927810 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.927846 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:33:31.927911 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:33:31.928034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":493,"threads_started":1}
I20260927 21:33:31.929800 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.919738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.3.186.195:44809)
0927 21:33:31.920912 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:31.920921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:31.920950 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:33:31.921030 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:31.921037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:31.921245 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:31.922118 (+ 873us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:31.922349 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.923849 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.926062 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.927397 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.929028 (+ 1631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.929243 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:31.929328 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:33:31.929454 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":843,"threads_started":1}
I20260927 21:33:31.931072 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.920392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55902)
0927 21:33:31.923500 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:31.923516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:31.923625 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:31.924150 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:31.924160 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:31.924188 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:31.924619 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.924630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.926419 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.926422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.928423 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:31.928440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.930392 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.930398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:31.930561 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.930588 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 21:33:31.930689 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:33:31.930795 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2974,"thread_start_us":2822,"threads_started":1}
I20260927 21:33:31.931205 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:31.920923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.3.186.193:39219)
0927 21:33:31.921455 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:31.921467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:31.923623 (+ 2156us) server_negotiation.cc:407] Connection header received
0927 21:33:31.923759 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:31.923768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:31.923970 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:31.924148 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:31.924802 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.926263 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.928661 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:31.929916 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:31.930050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:31.930225 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:31.930737 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 21:33:31.930904 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
I20260927 21:33:31.931951 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5337c59b049ff994f25ce1cfa4e36 (DEFAULT_TABLE table=test-table [id=9b61be4f286d46f286554cef1670ac79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:31.933284 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5337c59b049ff994f25ce1cfa4e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:31.933111 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5337c59b049ff994f25ce1cfa4e36 (DEFAULT_TABLE table=test-table [id=9b61be4f286d46f286554cef1670ac79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:31.934355 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5337c59b049ff994f25ce1cfa4e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:31.934589 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5337c59b049ff994f25ce1cfa4e36 (DEFAULT_TABLE table=test-table [id=9b61be4f286d46f286554cef1670ac79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:31.935808 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5337c59b049ff994f25ce1cfa4e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:31.950999 8396 tablet_bootstrap.cc:492] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Bootstrap starting.
I20260927 21:33:31.956856 8396 tablet_bootstrap.cc:654] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:31.957837 8395 tablet_bootstrap.cc:492] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Bootstrap starting.
I20260927 21:33:31.958899 8396 log.cc:824] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:31.961862 8395 tablet_bootstrap.cc:654] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:31.962276 8397 tablet_bootstrap.cc:492] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Bootstrap starting.
I20260927 21:33:31.962910 8112 heartbeater.cc:503] Master 127.3.186.254:45313 was elected leader, sending a full tablet report...
I20260927 21:33:31.963195 8395 log.cc:824] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:31.965219 8396 tablet_bootstrap.cc:492] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: No bootstrap required, opened a new log
I20260927 21:33:31.965560 8396 ts_tablet_manager.cc:1397] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260927 21:33:31.967100 8397 tablet_bootstrap.cc:654] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:31.968220 8397 log.cc:824] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:31.969255 8395 tablet_bootstrap.cc:492] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: No bootstrap required, opened a new log
I20260927 21:33:31.969508 8395 ts_tablet_manager.cc:1397] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20260927 21:33:31.970956 8397 tablet_bootstrap.cc:492] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: No bootstrap required, opened a new log
I20260927 21:33:31.971184 8397 ts_tablet_manager.cc:1397] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:33:31.973840 8396 raft_consensus.cc:348] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:31.974149 8396 raft_consensus.cc:374] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:31.974267 8396 raft_consensus.cc:729] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93e19841b244ee9b037724007eb5798, State: Initialized, Role: FOLLOWER
I20260927 21:33:31.974807 8396 consensus_queue.cc:260] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:31.975741 8377 heartbeater.cc:503] Master 127.3.186.254:45313 was elected leader, sending a full tablet report...
I20260927 21:33:31.976317 8396 ts_tablet_manager.cc:1428] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260927 21:33:31.979508 8395 raft_consensus.cc:348] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:31.979864 8395 raft_consensus.cc:374] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:31.979954 8395 raft_consensus.cc:729] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8744efe6708b4e7397f8ddc909a77943, State: Initialized, Role: FOLLOWER
I20260927 21:33:31.980480 8395 consensus_queue.cc:260] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:31.981036 8397 raft_consensus.cc:348] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:31.981487 8397 raft_consensus.cc:374] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:31.981630 8397 raft_consensus.cc:729] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdfa1cbb0d54a6881022cc0e7c66674, State: Initialized, Role: FOLLOWER
I20260927 21:33:31.982144 8245 heartbeater.cc:503] Master 127.3.186.254:45313 was elected leader, sending a full tablet report...
I20260927 21:33:31.982153 8397 consensus_queue.cc:260] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:31.982728 8395 ts_tablet_manager.cc:1428] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260927 21:33:31.983595 8397 ts_tablet_manager.cc:1428] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260927 21:33:32.047430 8403 raft_consensus.cc:482] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:32.047631 8403 raft_consensus.cc:504] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:32.049042 8403 leader_election.cc:288] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8744efe6708b4e7397f8ddc909a77943 (127.3.186.194:41601), d93e19841b244ee9b037724007eb5798 (127.3.186.195:44809)
I20260927 21:33:32.055406 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.049402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:33239 (local address 127.3.186.194:41601)
0927 21:33:32.049645 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:32.049654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:32.050081 (+ 427us) server_negotiation.cc:407] Connection header received
0927 21:33:32.050294 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:32.050320 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:32.050400 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:32.050492 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:32.051229 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.051944 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.053813 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.054552 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.054728 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.054794 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:32.055053 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:33:32.055173 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:33:32.055562 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.049379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:41601 (local address 127.3.186.193:33239)
0927 21:33:32.049963 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.049978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:32.050158 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:32.050642 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:32.050650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:32.050688 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:32.051086 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:32.051095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.052063 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.052071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:32.053665 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:32.053677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.054743 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.054749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:32.054894 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.054915 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:33:32.055030 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:33:32.055151 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"spinlock_wait_cycles":18688,"thread_start_us":332,"threads_started":1}
I20260927 21:33:32.055711 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.050351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:60531 (local address 127.3.186.195:44809)
0927 21:33:32.050540 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:32.050546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:32.050803 (+ 257us) server_negotiation.cc:407] Connection header received
0927 21:33:32.050995 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:32.051000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:32.051063 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:32.051165 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:32.051682 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.052360 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.054148 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.054860 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.055025 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.055092 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:32.055376 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:33:32.055488 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:33:32.055971 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.050088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:44809 (local address 127.3.186.193:60531)
0927 21:33:32.050704 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.050717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:32.050871 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:32.051177 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:32.051181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:32.051203 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:32.051523 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:32.051530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.052495 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.052500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:32.053987 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:32.054011 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.055058 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.055064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:32.055222 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.055243 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:33:32.055341 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:33:32.055455 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":314,"threads_started":1}
I20260927 21:33:32.056499 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5337c59b049ff994f25ce1cfa4e36" candidate_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8744efe6708b4e7397f8ddc909a77943" is_pre_election: true
I20260927 21:33:32.056838 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5337c59b049ff994f25ce1cfa4e36" candidate_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93e19841b244ee9b037724007eb5798" is_pre_election: true
I20260927 21:33:32.056946 8200 raft_consensus.cc:2454] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdfa1cbb0d54a6881022cc0e7c66674 in term 0.
I20260927 21:33:32.057268 8332 raft_consensus.cc:2454] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdfa1cbb0d54a6881022cc0e7c66674 in term 0.
I20260927 21:33:32.057615 8003 leader_election.cc:302] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdfa1cbb0d54a6881022cc0e7c66674, 8744efe6708b4e7397f8ddc909a77943; no voters:
I20260927 21:33:32.057936 8403 raft_consensus.cc:2781] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:32.058075 8403 raft_consensus.cc:482] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:32.058193 8403 raft_consensus.cc:3037] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:32.060134 8403 raft_consensus.cc:504] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:32.060608 8403 leader_election.cc:288] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [CANDIDATE]: Term 1 election: Requested vote from peers 8744efe6708b4e7397f8ddc909a77943 (127.3.186.194:41601), d93e19841b244ee9b037724007eb5798 (127.3.186.195:44809)
I20260927 21:33:32.060989 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5337c59b049ff994f25ce1cfa4e36" candidate_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8744efe6708b4e7397f8ddc909a77943"
I20260927 21:33:32.061051 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5337c59b049ff994f25ce1cfa4e36" candidate_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93e19841b244ee9b037724007eb5798"
I20260927 21:33:32.061151 8200 raft_consensus.cc:3037] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:32.061242 8332 raft_consensus.cc:3037] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:32.063235 8332 raft_consensus.cc:2454] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdfa1cbb0d54a6881022cc0e7c66674 in term 1.
I20260927 21:33:32.063238 8200 raft_consensus.cc:2454] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdfa1cbb0d54a6881022cc0e7c66674 in term 1.
I20260927 21:33:32.063596 8003 leader_election.cc:302] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdfa1cbb0d54a6881022cc0e7c66674, 8744efe6708b4e7397f8ddc909a77943; no voters:
I20260927 21:33:32.063855 8403 raft_consensus.cc:2781] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:32.064252 8403 raft_consensus.cc:686] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 1 LEADER]: Becoming Leader. State: Replica: 0cdfa1cbb0d54a6881022cc0e7c66674, State: Running, Role: LEADER
I20260927 21:33:32.064664 8403 consensus_queue.cc:237] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } }
I20260927 21:33:32.068770 7924 catalog_manager.cc:5495] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cdfa1cbb0d54a6881022cc0e7c66674 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdfa1cbb0d54a6881022cc0e7c66674" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39219 } health_report { overall_health: HEALTHY } } }
W20260927 21:33:32.109834 8378 tablet.cc:2369] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:32.116405 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.111345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.3.186.193:39219)
0927 21:33:32.111546 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:32.111552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:32.111667 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:33:32.111899 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:32.111903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:32.111976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:32.112111 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:32.112439 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.113078 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.114397 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.114769 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.114892 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.115369 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:32.115801 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:32.115950 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:32.116053 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:33:32.116122 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:32.123823 8332 raft_consensus.cc:1261] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 1 FOLLOWER]: Refusing update from remote peer 0cdfa1cbb0d54a6881022cc0e7c66674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:32.123823 8200 raft_consensus.cc:1261] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 1 FOLLOWER]: Refusing update from remote peer 0cdfa1cbb0d54a6881022cc0e7c66674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:32.124734 8406 consensus_queue.cc:1035] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8744efe6708b4e7397f8ddc909a77943" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:32.125056 8403 consensus_queue.cc:1035] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93e19841b244ee9b037724007eb5798" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:32.136139 8409 mvcc.cc:205] Tried to move back new op lower bound from 7334071550449123328 to 7334071550220091392. Current Snapshot: MvccSnapshot[applied={T|T < 7334071550449123328}]
I20260927 21:33:32.149670 7923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:32.149941 7923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:32.153399 7923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1dfb65e25c6443bac1b62dcb75c3321: Sending DeleteTablet for 3 replicas of tablet 51f5337c59b049ff994f25ce1cfa4e36
I20260927 21:33:32.154402 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5337c59b049ff994f25ce1cfa4e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:32 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:56934
I20260927 21:33:32.154553 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5337c59b049ff994f25ce1cfa4e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:32 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:55902
I20260927 21:33:32.154774 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5337c59b049ff994f25ce1cfa4e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:32 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:41542
I20260927 21:33:32.155247 8418 tablet_replica.cc:330] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: stopping tablet replica
I20260927 21:33:32.155550 8418 raft_consensus.cc:2229] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:32.155995 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7985
I20260927 21:33:32.156008 8418 raft_consensus.cc:2258] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:32.157737 8418 ts_tablet_manager.cc:1905] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:32.166114 8420 tablet_replica.cc:330] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: stopping tablet replica
I20260927 21:33:32.166263 8418 ts_tablet_manager.cc:1918] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:32.166476 8418 log.cc:1196] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/51f5337c59b049ff994f25ce1cfa4e36
I20260927 21:33:32.166653 8420 raft_consensus.cc:2229] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:32.167111 8418 ts_tablet_manager.cc:1939] T 51f5337c59b049ff994f25ce1cfa4e36 P 8744efe6708b4e7397f8ddc909a77943: Deleting consensus metadata
I20260927 21:33:32.167187 8420 raft_consensus.cc:2258] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:32.168903 7911 catalog_manager.cc:4841] TS 8744efe6708b4e7397f8ddc909a77943 (127.3.186.194:41601): tablet 51f5337c59b049ff994f25ce1cfa4e36 (table test-table [id=9b61be4f286d46f286554cef1670ac79]) successfully deleted
I20260927 21:33:32.169312 8420 ts_tablet_manager.cc:1905] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:32.177544 8420 ts_tablet_manager.cc:1918] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:32.177779 8420 log.cc:1196] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/51f5337c59b049ff994f25ce1cfa4e36
I20260927 21:33:32.177959 8419 tablet_replica.cc:330] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: stopping tablet replica
Sep 27 21:33:32 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:32.178330 8420 ts_tablet_manager.cc:1939] T 51f5337c59b049ff994f25ce1cfa4e36 P d93e19841b244ee9b037724007eb5798: Deleting consensus metadata
I20260927 21:33:32.178331 8419 raft_consensus.cc:2229] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:32.179006 8419 raft_consensus.cc:2258] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:32.179883 7911 catalog_manager.cc:4841] TS d93e19841b244ee9b037724007eb5798 (127.3.186.195:44809): tablet 51f5337c59b049ff994f25ce1cfa4e36 (table test-table [id=9b61be4f286d46f286554cef1670ac79]) successfully deleted
I20260927 21:33:32.181092 8419 ts_tablet_manager.cc:1905] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:32.186302 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.157277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.3.186.193:39219)
0927 21:33:32.157431 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:32.157436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:32.168200 (+ 10764us) server_negotiation.cc:407] Connection header received
0927 21:33:32.168383 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:32.168388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:32.168469 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:32.168568 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:32.170241 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.171466 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.173199 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.173901 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.173992 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.179222 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:32.179236 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:32.179248 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:32.179284 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:32.183228 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:32.184009 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:32.184016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:32.184020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:32.184187 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:32.184613 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:32.184617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:32.184620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:32.185026 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:32.185428 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:32.185677 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:33:32.185881 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:33:32.186949 8107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:32.279290 8419 ts_tablet_manager.cc:1918] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:32.279789 8419 log.cc:1196] T 51f5337c59b049ff994f25ce1cfa4e36 P 0cdfa1cbb0d54a6881022cc0e7c66674: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/51f5337c59b049ff994f25ce1cfa4e36
I20260927 21:33:32.279919 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7985
I20260927 21:33:32.297380 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8118
W20260927 21:33:32.297341 7911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:39219 (error 108)
I20260927 21:33:32.299103 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.298664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55930)
0927 21:33:32.298839 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.298943 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:33:32.299297 7911 catalog_manager.cc:4584] TS 0cdfa1cbb0d54a6881022cc0e7c66674 (127.3.186.193:39219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51f5337c59b049ff994f25ce1cfa4e36: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Sep 27 21:33:32 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:32.311213 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.298143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.3.186.194:41601)
0927 21:33:32.298314 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:32.298320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:32.298428 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:33:32.298653 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:32.298658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:32.298754 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:32.298844 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:32.300092 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.300809 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.301947 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.302322 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.302432 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.306170 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:32.306181 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:32.306200 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:32.306238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:32.308846 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:32.309494 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:32.309500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:32.309502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:32.309687 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:32.309998 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:32.310003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:32.310006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:32.310390 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:32.310580 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:32.310836 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:33:32.310996 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:33:32.311863 8240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:32.352563 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.352191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55934)
0927 21:33:32.352367 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.352438 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 21:33:32.353493 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.353109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55940)
0927 21:33:32.353266 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.353340 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:33:32.397624 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8118
I20260927 21:33:32.413614 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.413173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55948)
0927 21:33:32.413345 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.413414 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:33:32.414513 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.414201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55956)
0927 21:33:32.414341 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.414411 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:33:32.419876 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8250
Sep 27 21:33:32 dist-test-slave-pqr8 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:32.432835 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.420500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.3.186.195:44809)
0927 21:33:32.420655 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:32.420660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:32.420683 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:33:32.420829 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:32.420833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:32.420892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:32.420974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:32.422140 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.422816 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.423952 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:32.424302 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:32.424383 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:32.428125 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:32.428136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:32.428151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:32.428181 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:32.430703 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:32.431243 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:32.431248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:32.431250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:32.431373 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:32.431663 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:32.431666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:32.431668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:32.431984 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:32.432178 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:32.432390 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 21:33:32.432527 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:33:32.433458 8372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:32.495026 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.494631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55972)
0927 21:33:32.494823 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.494900 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 21:33:32.496004 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:32.495642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39219 (local address 127.0.0.1:55986)
0927 21:33:32.495846 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:32.495909 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 21:33:32.517585 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8250
I20260927 21:33:32.533149 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 7893
I20260927 21:33:32.534277 7954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:32.634651 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 7893
2026-09-27T21:33:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3864 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T21:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:32Z Disabled control of system clock
I20260927 21:33:32.696592 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42771
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:33:32.996884 8430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:32.997120 8430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:32.997197 8430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:33.003852 8430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:33.003934 8430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:33.003973 8430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:33.004038 8430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:33.004081 8430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:33.012380 8430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:42771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:33.013315 8430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:33.014811 8430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:33.025396 8436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:33.025465 8435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:33.025441 8438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:33.026671 8430 server_base.cc:1034] running on GCE node
I20260927 21:33:33.027374 8430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:33.028571 8430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:33.029785 8430 hybrid_clock.cc:648] HybridClock initialized: now 1790544813029745 us; error 45 us; skew 500 ppm
I20260927 21:33:33.030102 8430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:33.032094 8430 webserver.cc:460] Webserver started at http://127.3.186.254:40645/ using document root <none> and password file <none>
I20260927 21:33:33.032696 8430 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:33.032827 8430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:33.033120 8430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:33.035668 8430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "8fbf0bc9c6214d5497c723b15043e603"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "9df5a9aceb7bdfaae802d6e66785341a"
server_key_iv: "f991e8095e48d1b0ff58d98787927a33"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.036437 8430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "8fbf0bc9c6214d5497c723b15043e603"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "9df5a9aceb7bdfaae802d6e66785341a"
server_key_iv: "f991e8095e48d1b0ff58d98787927a33"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.041041 8430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 21:33:33.044119 8444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.045246 8430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:33.045429 8430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "8fbf0bc9c6214d5497c723b15043e603"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "9df5a9aceb7bdfaae802d6e66785341a"
server_key_iv: "f991e8095e48d1b0ff58d98787927a33"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.045662 8430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:33.052400 8430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:33.053340 8430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:33.053731 8430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:33.072625 8430 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:42771
I20260927 21:33:33.072659 8495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:42771 every 8 connection(s)
I20260927 21:33:33.073825 8430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:33.077085 8496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:33.079789 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8430
I20260927 21:33:33.079977 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:33.080592 3819 external_mini_cluster.cc:1428] Setting key b7df8386c151f580c228fccc4daf1e30
I20260927 21:33:33.088907 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:33.082454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.3.186.254:42771)
0927 21:33:33.083426 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:33.083439 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:33.083486 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:33:33.085192 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:33.085234 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:33.085788 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:33.086203 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:33.087164 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:33.087172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:33.087188 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:33.087203 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:33.087273 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:33.087294 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:33.087513 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:33.087647 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 21:33:33.087883 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":326,"threads_started":1}
I20260927 21:33:33.089709 8496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603: Bootstrap starting.
I20260927 21:33:33.092581 8496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:33.093416 8496 log.cc:824] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:33.095626 8496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603: No bootstrap required, opened a new log
I20260927 21:33:33.102480 8496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbf0bc9c6214d5497c723b15043e603" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42771 } }
I20260927 21:33:33.102792 8496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:33.102890 8496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbf0bc9c6214d5497c723b15043e603, State: Initialized, Role: FOLLOWER
I20260927 21:33:33.103374 8496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbf0bc9c6214d5497c723b15043e603" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42771 } }
I20260927 21:33:33.103609 8496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:33.103763 8496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:33.103925 8496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:33.106015 8496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbf0bc9c6214d5497c723b15043e603" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42771 } }
I20260927 21:33:33.106455 8496 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fbf0bc9c6214d5497c723b15043e603; no voters:
I20260927 21:33:33.106954 8496 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:33.107043 8501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:33.107750 8501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [term 1 LEADER]: Becoming Leader. State: Replica: 8fbf0bc9c6214d5497c723b15043e603, State: Running, Role: LEADER
I20260927 21:33:33.108209 8501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbf0bc9c6214d5497c723b15043e603" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42771 } }
I20260927 21:33:33.108912 8496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:33.111325 8503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fbf0bc9c6214d5497c723b15043e603. Latest consensus state: current_term: 1 leader_uuid: "8fbf0bc9c6214d5497c723b15043e603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbf0bc9c6214d5497c723b15043e603" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42771 } } }
I20260927 21:33:33.111754 8503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:33.111510 8502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fbf0bc9c6214d5497c723b15043e603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fbf0bc9c6214d5497c723b15043e603" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42771 } } }
I20260927 21:33:33.111929 8502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:33.124231 8507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:33.128152 8507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:33.134967 8507 catalog_manager.cc:1347] Generated new cluster ID: 484e14aac0fb48e49f0be85edb8b2401
I20260927 21:33:33.135067 8507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:33.146092 8507 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:33.146471 8507 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:33.161191 8507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603: Generated new TSK 0
I20260927 21:33:33.161995 8507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:33:33.169869 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42771
--builtin_ntp_servers=127.3.186.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:33:33.475548 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:33.475827 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:33.475972 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:33.482643 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:33.482764 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:33.482918 8520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:33.491245 8520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:33.492381 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:33.493989 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:33.506246 8526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:33.506209 8528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:33.506330 8525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:33.506903 8520 server_base.cc:1034] running on GCE node
I20260927 21:33:33.507642 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:33.508414 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:33.509615 8520 hybrid_clock.cc:648] HybridClock initialized: now 1790544813509574 us; error 34 us; skew 500 ppm
I20260927 21:33:33.509929 8520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:33.511914 8520 webserver.cc:460] Webserver started at http://127.3.186.193:44617/ using document root <none> and password file <none>
I20260927 21:33:33.512513 8520 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:33.512622 8520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:33.512928 8520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:33.515628 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "20d7f963322c44e2817e676f6651569c"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "cf7b6cf1ea75c3a6c5ed1a6deef66966"
server_key_iv: "1a6f40da152c812f709c6bae957d4b50"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.516316 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "20d7f963322c44e2817e676f6651569c"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "cf7b6cf1ea75c3a6c5ed1a6deef66966"
server_key_iv: "1a6f40da152c812f709c6bae957d4b50"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.520766 8520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 21:33:33.523795 8534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.524914 8520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:33:33.525068 8520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "20d7f963322c44e2817e676f6651569c"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "cf7b6cf1ea75c3a6c5ed1a6deef66966"
server_key_iv: "1a6f40da152c812f709c6bae957d4b50"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.525273 8520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:33.530342 8520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:33.531009 8520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:33.531255 8520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:33.532138 8520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:33.533360 8520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:33.533466 8520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.533609 8520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:33.533669 8520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.563548 8520 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:34853
I20260927 21:33:33.563621 8646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:34853 every 8 connection(s)
I20260927 21:33:33.564626 8520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:33.566176 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8520
I20260927 21:33:33.566398 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:33.567121 3819 external_mini_cluster.cc:1428] Setting key e55146dbc05fe98cefc73047c4dc434c
I20260927 21:33:33.569823 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42771
--builtin_ntp_servers=127.3.186.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:33:33.572373 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:33.567107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:52129 (local address 127.3.186.254:42771)
0927 21:33:33.567318 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:33.567324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:33.568619 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 21:33:33.570354 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:33.570359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:33.570417 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:33.570503 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:33.571685 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:33.571691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:33.571695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:33.571712 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:33.571751 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:33.571762 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:33.571882 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:33.572048 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:33:33.572108 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:33:33.573153 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:33.567519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42771 (local address 127.3.186.193:52129)
0927 21:33:33.568488 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:33.568520 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:33.570128 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:33.570698 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:33.570706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:33.570783 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:33.571426 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:33.571428 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:33.571460 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:33.571532 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:33.571850 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:33.571860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:33.571866 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:33:33.572119 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 21:33:33.572367 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
I20260927 21:33:33.574560 8647 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42771
I20260927 21:33:33.574824 8647 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:33.575374 8647 heartbeater.cc:511] Master 127.3.186.254:42771 requested a full tablet report, sending...
I20260927 21:33:33.577358 8461 ts_manager.cc:194] Registered new tserver with Master: 20d7f963322c44e2817e676f6651569c (127.3.186.193:34853)
W20260927 21:33:33.875314 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:33.875550 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:33.875667 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:33.882431 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:33.882507 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:33.882601 8650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:33.891084 8650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:33.892089 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:33.893632 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:33.906118 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:33.906160 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:33.906306 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:33.907001 8650 server_base.cc:1034] running on GCE node
I20260927 21:33:33.907550 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:33.908383 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:33.909667 8650 hybrid_clock.cc:648] HybridClock initialized: now 1790544813909603 us; error 39 us; skew 500 ppm
I20260927 21:33:33.909962 8650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:33.919059 8650 webserver.cc:460] Webserver started at http://127.3.186.194:39115/ using document root <none> and password file <none>
I20260927 21:33:33.919678 8650 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:33.919812 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:33.920099 8650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:33.922771 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "7556201711e845609f74ae214146e4f2"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "7834b512a5299c884fd4dd42a2c0eb78"
server_key_iv: "47e30057ed5043787707931d69c2a3ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.923542 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "7556201711e845609f74ae214146e4f2"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "7834b512a5299c884fd4dd42a2c0eb78"
server_key_iv: "47e30057ed5043787707931d69c2a3ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.928352 8650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:33:33.931583 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.932876 8650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:33:33.933048 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "7556201711e845609f74ae214146e4f2"
format_stamp: "Formatted at 2026-09-27 21:33:33 on dist-test-slave-pqr8"
server_key: "7834b512a5299c884fd4dd42a2c0eb78"
server_key_iv: "47e30057ed5043787707931d69c2a3ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:33.933259 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:33.938419 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:33.939086 8650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:33.939353 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:33.940196 8650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:33.941428 8650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:33.941601 8650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.941726 8650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:33.941813 8650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:33.973424 8650 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:39895
I20260927 21:33:33.973500 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:39895 every 8 connection(s)
I20260927 21:33:33.974670 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:33.976612 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8650
I20260927 21:33:33.976765 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:33.977305 3819 external_mini_cluster.cc:1428] Setting key 521e9f388f03b6a265fef76888eac152
I20260927 21:33:33.979795 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42771
--builtin_ntp_servers=127.3.186.212:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:33:33.982856 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:33.977246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:54237 (local address 127.3.186.254:42771)
0927 21:33:33.977462 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:33.977470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:33.978975 (+ 1505us) server_negotiation.cc:407] Connection header received
0927 21:33:33.980693 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:33.980698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:33.980766 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:33.980871 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:33.982091 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:33.982096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:33.982101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:33.982118 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:33.982157 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:33.982168 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:33.982277 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:33.982521 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 21:33:33.982584 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:33:33.983636 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:33.977783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42771 (local address 127.3.186.194:54237)
0927 21:33:33.978825 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:33.978864 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:33.980412 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:33.981081 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:33.981089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:33.981170 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:33.981826 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:33.981829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:33.981861 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:33.981935 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:33.982265 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:33.982274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:33.982281 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:33:33.982594 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 21:33:33.982808 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":335,"threads_started":1}
I20260927 21:33:33.984856 8777 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42771
I20260927 21:33:33.985150 8777 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:33.985766 8777 heartbeater.cc:511] Master 127.3.186.254:42771 requested a full tablet report, sending...
I20260927 21:33:33.986961 8461 ts_manager.cc:194] Registered new tserver with Master: 7556201711e845609f74ae214146e4f2 (127.3.186.194:39895)
W20260927 21:33:34.303767 8780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:34.304035 8780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:34.304183 8780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:34.311081 8780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:34.311196 8780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:34.311355 8780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:34.319909 8780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:34.320955 8780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:34.322623 8780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:34.334920 8785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:34.334944 8786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:34.335165 8788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:34.335553 8780 server_base.cc:1034] running on GCE node
I20260927 21:33:34.336180 8780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:34.337097 8780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:34.338325 8780 hybrid_clock.cc:648] HybridClock initialized: now 1790544814338287 us; error 29 us; skew 500 ppm
I20260927 21:33:34.338644 8780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:34.340672 8780 webserver.cc:460] Webserver started at http://127.3.186.195:42791/ using document root <none> and password file <none>
I20260927 21:33:34.341280 8780 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:34.341361 8780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:34.341745 8780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:34.344677 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "ec0d8c41599b4b058117407e80f15876"
format_stamp: "Formatted at 2026-09-27 21:33:34 on dist-test-slave-pqr8"
server_key: "04b8355092b59313ed11782c9c1182ba"
server_key_iv: "b1738fcf85d71ef6196ce8363d8a29d1"
server_key_version: "encryptionkey@0"
I20260927 21:33:34.345417 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "ec0d8c41599b4b058117407e80f15876"
format_stamp: "Formatted at 2026-09-27 21:33:34 on dist-test-slave-pqr8"
server_key: "04b8355092b59313ed11782c9c1182ba"
server_key_iv: "b1738fcf85d71ef6196ce8363d8a29d1"
server_key_version: "encryptionkey@0"
I20260927 21:33:34.350227 8780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:34.353193 8794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:34.354435 8780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:34.354597 8780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "ec0d8c41599b4b058117407e80f15876"
format_stamp: "Formatted at 2026-09-27 21:33:34 on dist-test-slave-pqr8"
server_key: "04b8355092b59313ed11782c9c1182ba"
server_key_iv: "b1738fcf85d71ef6196ce8363d8a29d1"
server_key_version: "encryptionkey@0"
I20260927 21:33:34.354807 8780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:34.360069 8780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:34.360726 8780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:34.360961 8780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:34.361829 8780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:34.363060 8780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:34.363178 8780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:34.363305 8780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:34.363389 8780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:34.394377 8780 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:36483
I20260927 21:33:34.394448 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:36483 every 8 connection(s)
I20260927 21:33:34.395486 8780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:34.396560 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8780
I20260927 21:33:34.396714 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:34.397271 3819 external_mini_cluster.cc:1428] Setting key 2e921f7ab89fb939c73b5206b63ba890
I20260927 21:33:34.403208 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.398019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:57287 (local address 127.3.186.254:42771)
0927 21:33:34.398209 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.398214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.399735 (+ 1521us) server_negotiation.cc:407] Connection header received
0927 21:33:34.401440 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.401445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.401527 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.401621 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.402699 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.402704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.402707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.402720 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.402748 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.402758 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.402845 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.402998 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.403039 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:33:34.403960 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.398589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42771 (local address 127.3.186.195:57287)
0927 21:33:34.399586 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.399624 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:34.401204 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:34.401804 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:34.401811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:34.401889 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:34.402462 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:34.402464 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:34.402495 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:34.402563 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:34.402835 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:34.402841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:34.402846 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:33:34.403034 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 21:33:34.403185 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":296,"threads_started":1}
I20260927 21:33:34.405057 8907 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42771
I20260927 21:33:34.405299 8907 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:34.405869 8907 heartbeater.cc:511] Master 127.3.186.254:42771 requested a full tablet report, sending...
I20260927 21:33:34.407030 8461 ts_manager.cc:194] Registered new tserver with Master: ec0d8c41599b4b058117407e80f15876 (127.3.186.195:36483)
I20260927 21:33:34.412076 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:33:34.418535 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.416401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.3.186.254:42771)
0927 21:33:34.416624 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.416632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.416967 (+ 335us) server_negotiation.cc:407] Connection header received
0927 21:33:34.417063 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.417067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.417128 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.417201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.418008 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.418012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.418015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.418027 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.418053 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.418063 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.418153 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.418215 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.418253 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:33:34.421875 8461 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:34.428843 8461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:34.446642 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.439203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:36483 (local address 127.0.0.1:43642)
0927 21:33:34.441149 (+ 1946us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.441187 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:34.441308 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:34.444449 (+ 3141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:34.444459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:34.444536 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:34.445460 (+ 924us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:34.445465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:34.445483 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:34.445559 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:34.445996 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:34.446003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:34.446014 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:33:34.446153 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:33:34.446207 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1709,"thread_start_us":2375,"threads_started":1}
I20260927 21:33:34.446651 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.440541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.3.186.193:34853)
0927 21:33:34.443354 (+ 2813us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.443362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.443386 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:33:34.443460 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.443468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.443684 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.443847 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.445580 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.445588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.445619 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.445640 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.445697 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.445722 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.445848 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.446252 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.446344 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2680,"thread_start_us":2885,"threads_started":1}
I20260927 21:33:34.447335 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.439299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.3.186.195:36483)
0927 21:33:34.443991 (+ 4692us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.443998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.444025 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:33:34.444086 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.444092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.444260 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.444403 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.445702 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.445711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.445721 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.445737 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.445786 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.445806 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.445951 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.446879 (+ 928us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.446953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4563,"thread_start_us":4406,"threads_started":1}
I20260927 21:33:34.447492 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.439541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55568)
0927 21:33:34.442454 (+ 2913us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.442470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:34.442562 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:34.444671 (+ 2109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:34.444678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:34.444705 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:34.445294 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:34.445297 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:34.445326 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:34.445388 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:34.445907 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:34.445919 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:34.445928 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:33:34.446104 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 21:33:34.447047 (+ 943us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2791,"mutex_wait_us":668,"thread_start_us":4516,"threads_started":1}
I20260927 21:33:34.448222 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.441149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:39895 (local address 127.0.0.1:52026)
0927 21:33:34.444290 (+ 3141us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.444303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:34.444411 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:34.445800 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:34.445804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:34.445828 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:34.447056 (+ 1228us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:34.447060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:34.447092 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:34.447135 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:34.447654 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:34.447661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:34.447669 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:33:34.447792 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 21:33:34.447843 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3034,"thread_start_us":1034,"threads_started":1}
I20260927 21:33:34.448910 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.440827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.3.186.194:39895)
0927 21:33:34.443015 (+ 2188us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.443031 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.444518 (+ 1487us) server_negotiation.cc:407] Connection header received
0927 21:33:34.444587 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.444597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.444786 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.444940 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.447309 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.447318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.447334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.447360 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.447413 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.447435 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.447598 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.448494 (+ 896us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.448581 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2046,"thread_start_us":1906,"threads_started":1}
I20260927 21:33:34.450541 8582 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3bc0074ccb4f01a18282a9de2a209a (DEFAULT_TABLE table=test-table [id=75a6e14d3fdf4b9cbb6866a17c4e629a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:34.450852 8842 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3bc0074ccb4f01a18282a9de2a209a (DEFAULT_TABLE table=test-table [id=75a6e14d3fdf4b9cbb6866a17c4e629a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:34.451664 8582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3bc0074ccb4f01a18282a9de2a209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:34.452144 8842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3bc0074ccb4f01a18282a9de2a209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:34.452991 8712 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3bc0074ccb4f01a18282a9de2a209a (DEFAULT_TABLE table=test-table [id=75a6e14d3fdf4b9cbb6866a17c4e629a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:34.454280 8712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3bc0074ccb4f01a18282a9de2a209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:34.462184 8925 tablet_bootstrap.cc:492] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: Bootstrap starting.
I20260927 21:33:34.465039 8925 tablet_bootstrap.cc:654] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:34.466033 8926 tablet_bootstrap.cc:492] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Bootstrap starting.
I20260927 21:33:34.466461 8925 log.cc:824] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:34.469265 8926 tablet_bootstrap.cc:654] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:34.469391 8925 tablet_bootstrap.cc:492] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: No bootstrap required, opened a new log
I20260927 21:33:34.469672 8925 ts_tablet_manager.cc:1397] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 21:33:34.470050 8928 tablet_bootstrap.cc:492] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Bootstrap starting.
I20260927 21:33:34.470419 8926 log.cc:824] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:34.473114 8926 tablet_bootstrap.cc:492] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: No bootstrap required, opened a new log
I20260927 21:33:34.473361 8926 ts_tablet_manager.cc:1397] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260927 21:33:34.473373 8928 tablet_bootstrap.cc:654] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:34.474527 8928 log.cc:824] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:34.477113 8928 tablet_bootstrap.cc:492] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: No bootstrap required, opened a new log
I20260927 21:33:34.477340 8928 ts_tablet_manager.cc:1397] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 21:33:34.477114 8925 raft_consensus.cc:348] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.477480 8925 raft_consensus.cc:374] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:34.477629 8925 raft_consensus.cc:729] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d7f963322c44e2817e676f6651569c, State: Initialized, Role: FOLLOWER
I20260927 21:33:34.478174 8925 consensus_queue.cc:260] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.482132 8647 heartbeater.cc:503] Master 127.3.186.254:42771 was elected leader, sending a full tablet report...
I20260927 21:33:34.482693 8925 ts_tablet_manager.cc:1428] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 21:33:34.482940 8926 raft_consensus.cc:348] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.483278 8926 raft_consensus.cc:374] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:34.483376 8926 raft_consensus.cc:729] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0d8c41599b4b058117407e80f15876, State: Initialized, Role: FOLLOWER
I20260927 21:33:34.483896 8926 consensus_queue.cc:260] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.485155 8907 heartbeater.cc:503] Master 127.3.186.254:42771 was elected leader, sending a full tablet report...
I20260927 21:33:34.485754 8926 ts_tablet_manager.cc:1428] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Time spent starting tablet: real 0.012s user 0.012s sys 0.002s
I20260927 21:33:34.487713 8928 raft_consensus.cc:348] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.488117 8928 raft_consensus.cc:374] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:34.488227 8928 raft_consensus.cc:729] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7556201711e845609f74ae214146e4f2, State: Initialized, Role: FOLLOWER
I20260927 21:33:34.488786 8928 consensus_queue.cc:260] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.489826 8777 heartbeater.cc:503] Master 127.3.186.254:42771 was elected leader, sending a full tablet report...
I20260927 21:33:34.490314 8928 ts_tablet_manager.cc:1428] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260927 21:33:34.566632 8648 tablet.cc:2369] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:34.579939 8932 raft_consensus.cc:482] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:34.580200 8932 raft_consensus.cc:504] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.581625 8932 leader_election.cc:288] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d7f963322c44e2817e676f6651569c (127.3.186.193:34853), 7556201711e845609f74ae214146e4f2 (127.3.186.194:39895)
I20260927 21:33:34.583940 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.581869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.3.186.195:48215)
0927 21:33:34.582071 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.582092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:34.582196 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:34.582577 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:34.582584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:34.582611 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:34.583210 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:34.583213 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:34.583230 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:34.583267 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:34.583582 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:34.583588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:34.583593 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:33:34.583705 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 21:33:34.583747 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 21:33:34.584051 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.581876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:48215 (local address 127.3.186.193:34853)
0927 21:33:34.582084 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.582102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.582172 (+ 70us) server_negotiation.cc:407] Connection header received
0927 21:33:34.582364 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.582371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.582433 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.582525 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.583403 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.583412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.583418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.583439 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.583477 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.583489 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.583570 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.583705 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.583763 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 21:33:34.584582 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.582173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:47643 (local address 127.3.186.194:39895)
0927 21:33:34.582359 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.582368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.582769 (+ 401us) server_negotiation.cc:407] Connection header received
0927 21:33:34.582928 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.582932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.582979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.583074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.583975 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.583982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.583987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.584005 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.584039 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.584055 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.584176 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.584331 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.584389 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:33:34.584827 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.582004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:39895 (local address 127.3.186.195:47643)
0927 21:33:34.582679 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.582692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:34.582806 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:34.583091 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:34.583096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:34.583116 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:34.583771 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:33:34.583774 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:34.583788 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:33:34.583826 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:34.584224 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:34.584228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:34.584232 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:33:34.584333 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:33:34.584373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":358,"threads_started":1}
I20260927 21:33:34.584849 8602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3bc0074ccb4f01a18282a9de2a209a" candidate_uuid: "ec0d8c41599b4b058117407e80f15876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7f963322c44e2817e676f6651569c" is_pre_election: true
I20260927 21:33:34.585366 8602 raft_consensus.cc:2454] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0d8c41599b4b058117407e80f15876 in term 0.
I20260927 21:33:34.585793 8732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3bc0074ccb4f01a18282a9de2a209a" candidate_uuid: "ec0d8c41599b4b058117407e80f15876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7556201711e845609f74ae214146e4f2" is_pre_election: true
I20260927 21:33:34.586025 8796 leader_election.cc:302] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d7f963322c44e2817e676f6651569c, ec0d8c41599b4b058117407e80f15876; no voters:
I20260927 21:33:34.586205 8732 raft_consensus.cc:2454] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec0d8c41599b4b058117407e80f15876 in term 0.
I20260927 21:33:34.586377 8932 raft_consensus.cc:2781] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:34.586546 8932 raft_consensus.cc:482] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:34.586670 8932 raft_consensus.cc:3037] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:34.588713 8932 raft_consensus.cc:504] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.589254 8932 leader_election.cc:288] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [CANDIDATE]: Term 1 election: Requested vote from peers 20d7f963322c44e2817e676f6651569c (127.3.186.193:34853), 7556201711e845609f74ae214146e4f2 (127.3.186.194:39895)
I20260927 21:33:34.589607 8602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3bc0074ccb4f01a18282a9de2a209a" candidate_uuid: "ec0d8c41599b4b058117407e80f15876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d7f963322c44e2817e676f6651569c"
I20260927 21:33:34.589686 8732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3bc0074ccb4f01a18282a9de2a209a" candidate_uuid: "ec0d8c41599b4b058117407e80f15876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7556201711e845609f74ae214146e4f2"
I20260927 21:33:34.589766 8602 raft_consensus.cc:3037] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:34.589893 8732 raft_consensus.cc:3037] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:34.592257 8602 raft_consensus.cc:2454] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0d8c41599b4b058117407e80f15876 in term 1.
I20260927 21:33:34.592288 8732 raft_consensus.cc:2454] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec0d8c41599b4b058117407e80f15876 in term 1.
I20260927 21:33:34.592669 8797 leader_election.cc:302] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7556201711e845609f74ae214146e4f2, ec0d8c41599b4b058117407e80f15876; no voters:
I20260927 21:33:34.592978 8932 raft_consensus.cc:2781] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:34.593370 8932 raft_consensus.cc:686] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 1 LEADER]: Becoming Leader. State: Replica: ec0d8c41599b4b058117407e80f15876, State: Running, Role: LEADER
I20260927 21:33:34.593782 8932 consensus_queue.cc:237] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } }
I20260927 21:33:34.597918 8461 catalog_manager.cc:5495] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec0d8c41599b4b058117407e80f15876 (127.3.186.195). New cstate: current_term: 1 leader_uuid: "ec0d8c41599b4b058117407e80f15876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec0d8c41599b4b058117407e80f15876" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 36483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:34.635780 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.633848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.3.186.195:36483)
0927 21:33:34.634036 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.634044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.634064 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:33:34.634280 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.634285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.634358 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.634458 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.635197 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.635203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.635207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.635223 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.635260 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.635272 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.635388 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.635508 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.635561 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:33:34.642894 8732 raft_consensus.cc:1261] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 1 FOLLOWER]: Refusing update from remote peer ec0d8c41599b4b058117407e80f15876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:34.642959 8602 raft_consensus.cc:1261] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c [term 1 FOLLOWER]: Refusing update from remote peer ec0d8c41599b4b058117407e80f15876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:34.643764 8935 consensus_queue.cc:1035] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d7f963322c44e2817e676f6651569c" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:34.644088 8932 consensus_queue.cc:1035] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7556201711e845609f74ae214146e4f2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:33:34.647032 8908 tablet.cc:2369] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:34.655035 8938 mvcc.cc:205] Tried to move back new op lower bound from 7334071560769236992 to 7334071560579604480. Current Snapshot: MvccSnapshot[applied={T|T < 7334071560769236992}]
I20260927 21:33:34.655207 8940 mvcc.cc:205] Tried to move back new op lower bound from 7334071560769236992 to 7334071560579604480. Current Snapshot: MvccSnapshot[applied={T|T < 7334071560769236992}]
I20260927 21:33:34.668200 8460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:34.668440 8460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:34.671470 8460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fbf0bc9c6214d5497c723b15043e603: Sending DeleteTablet for 3 replicas of tablet 3f3bc0074ccb4f01a18282a9de2a209a
I20260927 21:33:34.672405 8582 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3bc0074ccb4f01a18282a9de2a209a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:34 UTC) from {username='slave'} at 127.0.0.1:55568
I20260927 21:33:34.672384 8842 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3bc0074ccb4f01a18282a9de2a209a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:34 UTC) from {username='slave'} at 127.0.0.1:43642
I20260927 21:33:34.672600 8712 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3bc0074ccb4f01a18282a9de2a209a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:34 UTC) from {username='slave'} at 127.0.0.1:52026
I20260927 21:33:34.673908 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8520
I20260927 21:33:34.688951 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.675647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.3.186.193:34853)
0927 21:33:34.675853 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.675861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.685951 (+ 10090us) server_negotiation.cc:407] Connection header received
0927 21:33:34.686392 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.686400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.686534 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.686620 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.687976 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.687985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.687992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.688017 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.688062 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.688082 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.688193 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.688447 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.688530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:33:34.689195 8949 tablet_replica.cc:330] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: stopping tablet replica
I20260927 21:33:34.689779 8949 raft_consensus.cc:2229] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:34.690033 8642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:34.690558 8949 raft_consensus.cc:2258] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:34.697996 8950 tablet_replica.cc:330] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: stopping tablet replica
I20260927 21:33:34.698359 8950 raft_consensus.cc:2229] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:34.698849 8950 raft_consensus.cc:2258] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:34.699596 8949 ts_tablet_manager.cc:1905] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:34.700718 8950 ts_tablet_manager.cc:1905] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:34.706517 8949 ts_tablet_manager.cc:1918] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:34.707227 8949 log.cc:1196] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/3f3bc0074ccb4f01a18282a9de2a209a
I20260927 21:33:34.707811 8949 ts_tablet_manager.cc:1939] T 3f3bc0074ccb4f01a18282a9de2a209a P ec0d8c41599b4b058117407e80f15876: Deleting consensus metadata
I20260927 21:33:34.709764 8448 catalog_manager.cc:4841] TS ec0d8c41599b4b058117407e80f15876 (127.3.186.195:36483): tablet 3f3bc0074ccb4f01a18282a9de2a209a (table test-table [id=75a6e14d3fdf4b9cbb6866a17c4e629a]) successfully deleted
I20260927 21:33:34.710210 8950 ts_tablet_manager.cc:1918] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:34.710352 8950 log.cc:1196] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/3f3bc0074ccb4f01a18282a9de2a209a
I20260927 21:33:34.710762 8950 ts_tablet_manager.cc:1939] T 3f3bc0074ccb4f01a18282a9de2a209a P 7556201711e845609f74ae214146e4f2: Deleting consensus metadata
I20260927 21:33:34.711992 8447 catalog_manager.cc:4841] TS 7556201711e845609f74ae214146e4f2 (127.3.186.194:39895): tablet 3f3bc0074ccb4f01a18282a9de2a209a (table test-table [id=75a6e14d3fdf4b9cbb6866a17c4e629a]) successfully deleted
I20260927 21:33:34.809499 8948 tablet_replica.cc:330] T 3f3bc0074ccb4f01a18282a9de2a209a P 20d7f963322c44e2817e676f6651569c: stopping tablet replica
I20260927 21:33:34.809827 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8520
I20260927 21:33:34.832018 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8650
W20260927 21:33:34.832005 8446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:34853 (error 108)
I20260927 21:33:34.834997 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.834530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55580)
0927 21:33:34.834706 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.834833 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:33:34.835217 8446 catalog_manager.cc:4584] TS 20d7f963322c44e2817e676f6651569c (127.3.186.193:34853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f3bc0074ccb4f01a18282a9de2a209a: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
I20260927 21:33:34.835383 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.832777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.3.186.194:39895)
0927 21:33:34.832951 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.832958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.833234 (+ 276us) server_negotiation.cc:407] Connection header received
0927 21:33:34.833444 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.833451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.833588 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.833690 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.834762 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.834769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.834773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.834791 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.834831 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.834846 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.834976 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.835061 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.835119 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:33:34.836037 8772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:34.888613 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.888196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55596)
0927 21:33:34.888384 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.888468 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 21:33:34.889564 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.889219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55602)
0927 21:33:34.889349 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.889425 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:33:34.949710 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8650
I20260927 21:33:34.949934 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.949429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55618)
0927 21:33:34.949650 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.949763 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 21:33:34.951889 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.951513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55626)
0927 21:33:34.951664 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:34.951746 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:33:34.966337 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8780
I20260927 21:33:34.969236 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:34.967045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.3.186.195:36483)
0927 21:33:34.967242 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:34.967249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:34.967311 (+ 62us) server_negotiation.cc:407] Connection header received
0927 21:33:34.967528 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:34.967533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:34.967593 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:34.967705 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:34.968580 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:34.968585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:34.968590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:34.968606 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:34.968647 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:34.968659 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:34.968806 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:34.968892 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:33:34.968943 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:33:34.969980 8902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:35.032078 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:35.031674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55636)
0927 21:33:35.031853 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:35.031932 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:33:35.033042 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:35.032691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34853 (local address 127.0.0.1:55646)
0927 21:33:35.032862 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:35.032928 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:33:35.083758 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8780
I20260927 21:33:35.101831 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8430
I20260927 21:33:35.102828 8491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:35.201738 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8430
2026-09-27T21:33:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2566 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 21:33:35.241863 3819 mini_oidc.cc:152] Starting JWKS server
I20260927 21:33:35.244313 3819 webserver.cc:460] Webserver started at https://127.0.0.1:33015/ using document root <none> and password file <none>
I20260927 21:33:35.244671 3819 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:33:35.245267 3819 webserver.cc:460] Webserver started at http://127.0.0.1:32827/ using document root <none> and password file <none>
2026-09-27T21:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:35Z Disabled control of system clock
I20260927 21:33:35.268581 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42261
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:42261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/testchainca.cert with env {}
W20260927 21:33:35.563227 8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:35.563467 8962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:35.563544 8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:35.570102 8962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:35.570182 8962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:35.570225 8962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:35.570278 8962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:35.570319 8962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:33:35.570369 8962 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:33:35.570405 8962 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33015/jwks.json
I20260927 21:33:35.579413 8962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:42261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:35.580423 8962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:35.581653 8962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:35.592636 8967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:35.592612 8968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:35.592720 8970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:35.593665 8962 server_base.cc:1034] running on GCE node
I20260927 21:33:35.594416 8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:35.595585 8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:35.596798 8962 hybrid_clock.cc:648] HybridClock initialized: now 1790544815596766 us; error 25 us; skew 500 ppm
I20260927 21:33:35.597136 8962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:35.599241 8962 webserver.cc:460] Webserver started at http://127.3.186.254:40525/ using document root <none> and password file <none>
I20260927 21:33:35.599843 8962 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:35.599969 8962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:35.600260 8962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:35.602874 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "12d0ad9ab3514a019b36d77f20169c8b"
format_stamp: "Formatted at 2026-09-27 21:33:35 on dist-test-slave-pqr8"
server_key: "b5f15f2fc0f43a698a04c1c7fd97ef32"
server_key_iv: "2c2f3bd157775acd56c2d2dd590b3f79"
server_key_version: "encryptionkey@0"
I20260927 21:33:35.603628 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "12d0ad9ab3514a019b36d77f20169c8b"
format_stamp: "Formatted at 2026-09-27 21:33:35 on dist-test-slave-pqr8"
server_key: "b5f15f2fc0f43a698a04c1c7fd97ef32"
server_key_iv: "2c2f3bd157775acd56c2d2dd590b3f79"
server_key_version: "encryptionkey@0"
I20260927 21:33:35.608073 8962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:33:35.611030 8976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:35.612183 8962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:35.612349 8962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "12d0ad9ab3514a019b36d77f20169c8b"
format_stamp: "Formatted at 2026-09-27 21:33:35 on dist-test-slave-pqr8"
server_key: "b5f15f2fc0f43a698a04c1c7fd97ef32"
server_key_iv: "2c2f3bd157775acd56c2d2dd590b3f79"
server_key_version: "encryptionkey@0"
I20260927 21:33:35.612538 8962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:35.664860 8962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:35.665578 8962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:35.665856 8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:35.682555 8962 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:42261
I20260927 21:33:35.682624 9027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:42261 every 8 connection(s)
I20260927 21:33:35.683720 8962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:35.686933 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:35.692368 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 8962
I20260927 21:33:35.692574 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:35.693312 3819 external_mini_cluster.cc:1428] Setting key 9fdb7505eade1043a02eebedd7bdc518
I20260927 21:33:35.697175 9028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b: Bootstrap starting.
I20260927 21:33:35.700009 9028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:35.700910 9028 log.cc:824] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:35.703382 9028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b: No bootstrap required, opened a new log
I20260927 21:33:35.705591 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:35.695120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.3.186.254:42261)
0927 21:33:35.695954 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:35.695964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:35.695999 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:33:35.697741 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:35.697761 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:35.698150 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:35.698521 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:35.699038 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:35.700257 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:35.701591 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:35.702199 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:35.702381 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:35.703468 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:35.703496 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:35.703513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:35.704013 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:35.704084 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:35.704102 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:35.704224 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:35.704432 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 21:33:35.704690 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260927 21:33:35.709079 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:33:35.711493 9028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d0ad9ab3514a019b36d77f20169c8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42261 } }
I20260927 21:33:35.711866 9028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:35.711985 9028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d0ad9ab3514a019b36d77f20169c8b, State: Initialized, Role: FOLLOWER
I20260927 21:33:35.712464 9028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d0ad9ab3514a019b36d77f20169c8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42261 } }
I20260927 21:33:35.712666 9028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:35.712778 9028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:35.712927 9028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:35.714960 9028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d0ad9ab3514a019b36d77f20169c8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42261 } }
I20260927 21:33:35.715517 9028 leader_election.cc:302] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12d0ad9ab3514a019b36d77f20169c8b; no voters:
I20260927 21:33:35.716019 9028 leader_election.cc:288] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:35.716193 9033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:35.716962 9033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [term 1 LEADER]: Becoming Leader. State: Replica: 12d0ad9ab3514a019b36d77f20169c8b, State: Running, Role: LEADER
I20260927 21:33:35.717451 9033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d0ad9ab3514a019b36d77f20169c8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42261 } }
I20260927 21:33:35.717965 9028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:35.720201 9034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12d0ad9ab3514a019b36d77f20169c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d0ad9ab3514a019b36d77f20169c8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42261 } } }
I20260927 21:33:35.720671 9034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:35.720227 9035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d0ad9ab3514a019b36d77f20169c8b. Latest consensus state: current_term: 1 leader_uuid: "12d0ad9ab3514a019b36d77f20169c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d0ad9ab3514a019b36d77f20169c8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42261 } } }
I20260927 21:33:35.721329 9035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:35.722795 9041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:35.727293 9041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:35.734442 9041 catalog_manager.cc:1347] Generated new cluster ID: f14d2d56c23c4921887fe6c7f61bc9ad
I20260927 21:33:35.734539 9041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:35.743067 9041 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:35.744618 9041 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:35.756129 9041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12d0ad9ab3514a019b36d77f20169c8b: Generated new TSK 0
I20260927 21:33:35.756896 9041 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:33:35.804366 9058 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 21:33:35.804507 9058 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:33:35.805348 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:35.776338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.3.186.254:42261)
0927 21:33:35.776514 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:35.776528 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:35.777010 (+ 482us) server_negotiation.cc:407] Connection header received
0927 21:33:35.777233 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:35.777238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:35.777342 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:35.777485 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:33:35.777994 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:35.779087 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:35.780446 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:35.781127 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:35.781256 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:35.781721 (+ 465us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:33:35.804748 (+ 23027us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:33:35.805062 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 21:33:35.805455 9031 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 21:33:35.806761 9057 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:35.776279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42261 (local address 127.0.0.1:35030)
0927 21:33:35.776919 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:35.776933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:35.777051 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:35.777522 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:35.777526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:35.777535 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:33:35.777855 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:35.777863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:35.779224 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:35.779227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:35.780308 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:35.780318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:35.781280 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:35.781285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:35.781403 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:35.781449 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:33:35.804978 (+ 23529us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:33:35.805269 (+ 291us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.186.254:42261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":358,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.186.254:42261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 21:33:35.808184 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 8962
I20260927 21:33:35.809226 9023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:35.895745 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 8962
2026-09-27T21:33:35Z chronyd exiting
I20260927 21:33:35.945159 3819 test_util.cc:175] -----------------------------------------------
I20260927 21:33:35.945277 3819 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544773327888-3819-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (707 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 21:33:35.948819 3819 mini_oidc.cc:152] Starting JWKS server
I20260927 21:33:35.950555 3819 webserver.cc:460] Webserver started at https://127.0.0.1:38563/ using document root <none> and password file <none>
I20260927 21:33:35.950719 3819 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:33:35.951254 3819 webserver.cc:460] Webserver started at http://127.0.0.1:35591/ using document root <none> and password file <none>
2026-09-27T21:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:35Z Disabled control of system clock
I20260927 21:33:35.974253 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42111
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:34879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:42111
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38563/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/testchainca.cert with env {}
W20260927 21:33:36.265890 9065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:36.266155 9065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:36.266307 9065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:36.273006 9065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:36.273128 9065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:36.273221 9065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:36.273320 9065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:36.273402 9065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:33:36.273486 9065 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:33:36.273579 9065 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38563/jwks.json
I20260927 21:33:36.282351 9065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:34879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:42111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38563/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:36.283421 9065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:36.284583 9065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:36.295265 9073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:36.295320 9071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:36.295265 9070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:36.296103 9065 server_base.cc:1034] running on GCE node
I20260927 21:33:36.296805 9065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:36.298080 9065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:36.299302 9065 hybrid_clock.cc:648] HybridClock initialized: now 1790544816299277 us; error 43 us; skew 500 ppm
I20260927 21:33:36.299614 9065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:36.301724 9065 webserver.cc:460] Webserver started at http://127.3.186.254:44101/ using document root <none> and password file <none>
I20260927 21:33:36.302306 9065 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:36.302448 9065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:36.302781 9065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:36.305291 9065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "665c4abf5d3640e9a4db612b8acc682f"
format_stamp: "Formatted at 2026-09-27 21:33:36 on dist-test-slave-pqr8"
server_key: "1642425d986d5ce6801291dff10dc136"
server_key_iv: "daf9c351582d34d7e25bf9ea9376d6db"
server_key_version: "encryptionkey@0"
I20260927 21:33:36.306103 9065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "665c4abf5d3640e9a4db612b8acc682f"
format_stamp: "Formatted at 2026-09-27 21:33:36 on dist-test-slave-pqr8"
server_key: "1642425d986d5ce6801291dff10dc136"
server_key_iv: "daf9c351582d34d7e25bf9ea9376d6db"
server_key_version: "encryptionkey@0"
I20260927 21:33:36.310750 9065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 21:33:36.313868 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:36.315050 9065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 21:33:36.315222 9065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "665c4abf5d3640e9a4db612b8acc682f"
format_stamp: "Formatted at 2026-09-27 21:33:36 on dist-test-slave-pqr8"
server_key: "1642425d986d5ce6801291dff10dc136"
server_key_iv: "daf9c351582d34d7e25bf9ea9376d6db"
server_key_version: "encryptionkey@0"
I20260927 21:33:36.315418 9065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:36.332491 9065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:36.333151 9065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:36.333434 9065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:36.350567 9065 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:42111
I20260927 21:33:36.350591 9130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:42111 every 8 connection(s)
I20260927 21:33:36.351770 9065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:36.354914 9131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:36.357654 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9065
I20260927 21:33:36.357849 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:36.358603 3819 external_mini_cluster.cc:1428] Setting key 3c686877b24776ccaa38bbf5db27eb1c
I20260927 21:33:36.365859 9131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f: Bootstrap starting.
I20260927 21:33:36.368733 9131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:36.369823 9131 log.cc:824] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:36.370839 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:36.360255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.3.186.254:42111)
0927 21:33:36.361078 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:36.361087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:36.361129 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:33:36.362817 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:36.362836 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:36.363245 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:36.363589 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:36.364095 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:36.365230 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:36.366611 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:36.367241 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:36.367449 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:36.368539 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:36.368545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:36.368558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:36.369073 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:36.369169 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:36.369188 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:36.369304 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:36.369530 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 21:33:36.369792 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":319,"threads_started":1}
I20260927 21:33:36.372551 9131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f: No bootstrap required, opened a new log
I20260927 21:33:36.374210 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:33:36.379791 9131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665c4abf5d3640e9a4db612b8acc682f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42111 } }
I20260927 21:33:36.380084 9131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:36.380189 9131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665c4abf5d3640e9a4db612b8acc682f, State: Initialized, Role: FOLLOWER
I20260927 21:33:36.380687 9131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665c4abf5d3640e9a4db612b8acc682f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42111 } }
I20260927 21:33:36.380894 9131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:36.381035 9131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:36.381191 9131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:36.383253 9131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665c4abf5d3640e9a4db612b8acc682f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42111 } }
I20260927 21:33:36.383695 9131 leader_election.cc:302] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665c4abf5d3640e9a4db612b8acc682f; no voters:
I20260927 21:33:36.384179 9131 leader_election.cc:288] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:36.384306 9136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:36.385311 9136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [term 1 LEADER]: Becoming Leader. State: Replica: 665c4abf5d3640e9a4db612b8acc682f, State: Running, Role: LEADER
I20260927 21:33:36.385847 9136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665c4abf5d3640e9a4db612b8acc682f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42111 } }
I20260927 21:33:36.385990 9131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:36.388307 9138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665c4abf5d3640e9a4db612b8acc682f. Latest consensus state: current_term: 1 leader_uuid: "665c4abf5d3640e9a4db612b8acc682f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665c4abf5d3640e9a4db612b8acc682f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42111 } } }
I20260927 21:33:36.388383 9137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665c4abf5d3640e9a4db612b8acc682f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665c4abf5d3640e9a4db612b8acc682f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42111 } } }
I20260927 21:33:36.388710 9138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:36.388757 9137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:36.392289 9145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:36.396266 9145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:36.403132 9145 catalog_manager.cc:1347] Generated new cluster ID: 23c36d1d1b3041da8b07f55d147d5e18
I20260927 21:33:36.403237 9145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:36.420272 9145 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:36.421368 9145 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:36.432495 9145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665c4abf5d3640e9a4db612b8acc682f: Generated new TSK 0
I20260927 21:33:36.433179 9145 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:33:36.471725 9161 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 21:33:36.471822 9161 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:33:36.472929 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:36.453607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.3.186.254:42111)
0927 21:33:36.453818 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:36.453827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:36.454204 (+ 377us) server_negotiation.cc:407] Connection header received
0927 21:33:36.454404 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:36.454407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:36.454462 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:36.454562 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:33:36.455043 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:36.455709 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:36.457672 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:36.458089 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:36.458177 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:36.458694 (+ 517us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:33:36.472304 (+ 13610us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38563/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:33:36.472529 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38563/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 21:33:36.473086 9160 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:36.453474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42111 (local address 127.0.0.1:44668)
0927 21:33:36.454100 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:36.454118 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:36.454256 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:36.454602 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:36.454607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:36.454620 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:33:36.454890 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:36.454901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:36.455833 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:36.455839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:36.457548 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:36.457563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:36.458240 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:36.458249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:36.458450 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:36.458493 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:33:36.472483 (+ 13990us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38563/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:33:36.472717 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.186.254:42111: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38563/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":359,"threads_started":1}
W20260927 21:33:36.473089 9134 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38563/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 21:33:36.474331 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9065
I20260927 21:33:36.475260 9126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:36.562870 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9065
2026-09-27T21:33:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (669 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 21:33:36.617909 3819 mini_oidc.cc:152] Starting JWKS server
I20260927 21:33:36.619653 3819 webserver.cc:460] Webserver started at https://127.0.0.1:36027/ using document root <none> and password file <none>
I20260927 21:33:36.619867 3819 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:33:36.620410 3819 webserver.cc:460] Webserver started at http://127.0.0.1:38775/ using document root <none> and password file <none>
2026-09-27T21:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:36Z Disabled control of system clock
I20260927 21:33:36.643595 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:37605
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:37605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:33:36.940081 9168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:36.940356 9168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:36.940503 9168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:36.947288 9168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:36.947453 9168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:36.947559 9168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:36.947656 9168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:36.947732 9168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:33:36.947814 9168 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:33:36.947890 9168 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36027/jwks.json
I20260927 21:33:36.956199 9168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:37605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:37605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:36.957242 9168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:36.958863 9168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:36.969458 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:36.969420 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:36.969488 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:36.970271 9168 server_base.cc:1034] running on GCE node
I20260927 21:33:36.970953 9168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:36.972131 9168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:36.973309 9168 hybrid_clock.cc:648] HybridClock initialized: now 1790544816973272 us; error 39 us; skew 500 ppm
I20260927 21:33:36.973620 9168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:36.981045 9168 webserver.cc:460] Webserver started at http://127.3.186.254:45425/ using document root <none> and password file <none>
I20260927 21:33:36.981734 9168 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:36.981839 9168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:36.982136 9168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:36.984582 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "7574e7c1617b4a0397533424490986f3"
format_stamp: "Formatted at 2026-09-27 21:33:36 on dist-test-slave-pqr8"
server_key: "4492e51e9b526ee7d84f82d87ac2929a"
server_key_iv: "323020f2ae52fd9fe41bd03b9d6e5623"
server_key_version: "encryptionkey@0"
I20260927 21:33:36.985314 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "7574e7c1617b4a0397533424490986f3"
format_stamp: "Formatted at 2026-09-27 21:33:36 on dist-test-slave-pqr8"
server_key: "4492e51e9b526ee7d84f82d87ac2929a"
server_key_iv: "323020f2ae52fd9fe41bd03b9d6e5623"
server_key_version: "encryptionkey@0"
I20260927 21:33:36.989873 9168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260927 21:33:36.992743 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:36.993808 9168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:36.993973 9168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "7574e7c1617b4a0397533424490986f3"
format_stamp: "Formatted at 2026-09-27 21:33:36 on dist-test-slave-pqr8"
server_key: "4492e51e9b526ee7d84f82d87ac2929a"
server_key_iv: "323020f2ae52fd9fe41bd03b9d6e5623"
server_key_version: "encryptionkey@0"
I20260927 21:33:36.994165 9168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:37.019909 9168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:37.020588 9168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:37.020862 9168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:37.037477 9168 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:37605
I20260927 21:33:37.037540 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:37605 every 8 connection(s)
I20260927 21:33:37.038671 9168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:37.041915 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:37.047199 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9168
I20260927 21:33:37.047405 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:37.047987 3819 external_mini_cluster.cc:1428] Setting key 6eb8cf34b17844cdf265a8f250e8b8b0
I20260927 21:33:37.051692 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3: Bootstrap starting.
I20260927 21:33:37.055279 9234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:37.056445 9234 log.cc:824] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:37.059302 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3: No bootstrap required, opened a new log
I20260927 21:33:37.062561 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:37.049720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.3.186.254:37605)
0927 21:33:37.050607 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:37.050620 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:37.050663 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:33:37.052895 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:37.052919 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:37.053489 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:37.053894 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:37.054328 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.056225 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.057895 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.058878 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.059037 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:37.060012 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:37.060024 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:37.060044 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:37.060669 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:37.060757 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:37.060780 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:37.060890 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:37.061099 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 21:33:37.061408 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":277,"threads_started":1}
I20260927 21:33:37.066099 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:33:37.067585 9234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7574e7c1617b4a0397533424490986f3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37605 } }
I20260927 21:33:37.067888 9234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:37.067999 9234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7574e7c1617b4a0397533424490986f3, State: Initialized, Role: FOLLOWER
I20260927 21:33:37.068491 9234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7574e7c1617b4a0397533424490986f3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37605 } }
I20260927 21:33:37.068724 9234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:37.068867 9234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:37.069036 9234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:37.070922 9234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7574e7c1617b4a0397533424490986f3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37605 } }
I20260927 21:33:37.071332 9234 leader_election.cc:302] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7574e7c1617b4a0397533424490986f3; no voters:
I20260927 21:33:37.071859 9234 leader_election.cc:288] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:37.071991 9239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:37.072803 9239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7574e7c1617b4a0397533424490986f3, State: Running, Role: LEADER
I20260927 21:33:37.073273 9239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7574e7c1617b4a0397533424490986f3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37605 } }
I20260927 21:33:37.073743 9234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:37.075781 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7574e7c1617b4a0397533424490986f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7574e7c1617b4a0397533424490986f3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37605 } } }
I20260927 21:33:37.076117 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:37.076614 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7574e7c1617b4a0397533424490986f3. Latest consensus state: current_term: 1 leader_uuid: "7574e7c1617b4a0397533424490986f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7574e7c1617b4a0397533424490986f3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37605 } } }
I20260927 21:33:37.076946 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:37.077013 9248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:37.082298 9248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:37.089720 9248 catalog_manager.cc:1347] Generated new cluster ID: e446051a1beb43c5acad0c3420de3a43
I20260927 21:33:37.089814 9248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:37.108161 9248 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:37.109758 9248 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:37.117815 9248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7574e7c1617b4a0397533424490986f3: Generated new TSK 0
I20260927 21:33:37.118407 9248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:33:37.136654 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:37.129022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.3.186.254:37605)
0927 21:33:37.129182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:37.129188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:37.129560 (+ 372us) server_negotiation.cc:407] Connection header received
0927 21:33:37.129774 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:37.129780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:37.129865 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:37.130011 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:33:37.130369 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.131437 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.132738 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.133416 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.133566 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:37.133958 (+ 392us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:33:37.136082 (+ 2124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:33:37.136365 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 21:33:37.136760 9237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 21:33:37.138458 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:37.128915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:37605 (local address 127.0.0.1:42940)
0927 21:33:37.129454 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:37.129468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:37.129628 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:37.130012 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:37.130015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:37.130023 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:33:37.130242 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:37.130248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.131552 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.131560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:37.132599 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:37.132609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.133616 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.133620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:37.133732 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:37.133759 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:33:37.136258 (+ 2499us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:33:37.138045 (+ 1787us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.186.254:37605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36027/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1528,"thread_start_us":291,"threads_started":1}
I20260927 21:33:37.139323 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9168
I20260927 21:33:37.140201 9229 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 21:33:37.147742 9264 webserver.cc:533] Webserver: sslize failed:
E20260927 21:33:37.147845 9264 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:33:37.224690 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9168
2026-09-27T21:33:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (661 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T21:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:37Z Disabled control of system clock
I20260927 21:33:37.301872 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43663
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:43663
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/testchainca.cert with env {}
W20260927 21:33:37.598302 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:37.598553 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:37.598631 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:37.605144 9269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:37.605227 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:37.605262 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:37.605315 9269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:37.605353 9269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:33:37.605409 9269 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:33:37.605448 9269 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 21:33:37.614079 9269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:43663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:37.615113 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:37.616163 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:37.626974 9274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:37.626937 9275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:37.627060 9277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:37.627629 9269 server_base.cc:1034] running on GCE node
I20260927 21:33:37.628324 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:37.629462 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:37.630704 9269 hybrid_clock.cc:648] HybridClock initialized: now 1790544817630667 us; error 36 us; skew 500 ppm
I20260927 21:33:37.631031 9269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:33:37.633018 9269 webserver.cc:460] Webserver started at http://127.3.186.254:38621/ using document root <none> and password file <none>
I20260927 21:33:37.633678 9269 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:37.633786 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:37.634093 9269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:37.636572 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "e594bb08ed9b4e009f02a139967b54a1"
format_stamp: "Formatted at 2026-09-27 21:33:37 on dist-test-slave-pqr8"
server_key: "cc0b7a23e9170c484f108d897f31a295"
server_key_iv: "aa91b8add04c2e6a511f4ccd07ac2b4c"
server_key_version: "encryptionkey@0"
I20260927 21:33:37.637295 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "e594bb08ed9b4e009f02a139967b54a1"
format_stamp: "Formatted at 2026-09-27 21:33:37 on dist-test-slave-pqr8"
server_key: "cc0b7a23e9170c484f108d897f31a295"
server_key_iv: "aa91b8add04c2e6a511f4ccd07ac2b4c"
server_key_version: "encryptionkey@0"
I20260927 21:33:37.641675 9269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:37.644652 9283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:37.645973 9269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:37.646139 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "e594bb08ed9b4e009f02a139967b54a1"
format_stamp: "Formatted at 2026-09-27 21:33:37 on dist-test-slave-pqr8"
server_key: "cc0b7a23e9170c484f108d897f31a295"
server_key_iv: "aa91b8add04c2e6a511f4ccd07ac2b4c"
server_key_version: "encryptionkey@0"
I20260927 21:33:37.646329 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:37.659588 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:37.660249 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:37.660495 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:37.677318 9269 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:43663
I20260927 21:33:37.677335 9334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:43663 every 8 connection(s)
I20260927 21:33:37.678459 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:37.681656 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:37.685814 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9269
I20260927 21:33:37.685997 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:37.686780 3819 external_mini_cluster.cc:1428] Setting key e6215009c33d2662653aa7a3551b88bf
I20260927 21:33:37.691891 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1: Bootstrap starting.
I20260927 21:33:37.695071 9335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:37.696110 9335 log.cc:824] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:37.698927 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1: No bootstrap required, opened a new log
I20260927 21:33:37.699324 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:37.688469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.3.186.254:43663)
0927 21:33:37.689384 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:37.689396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:37.689437 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:33:37.691008 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:37.691026 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:37.691425 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:37.691699 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:37.692236 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.693472 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.695278 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.695913 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.696036 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:37.697241 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:37.697254 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:37.697282 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:33:37.697824 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:37.697890 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:33:37.697912 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:37.697992 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:37.698211 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:33:37.698501 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":347,"threads_started":1}
I20260927 21:33:37.702540 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:33:37.706861 9335 raft_consensus.cc:348] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e594bb08ed9b4e009f02a139967b54a1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43663 } }
I20260927 21:33:37.707115 9335 raft_consensus.cc:374] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:37.707182 9335 raft_consensus.cc:729] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e594bb08ed9b4e009f02a139967b54a1, State: Initialized, Role: FOLLOWER
I20260927 21:33:37.707614 9335 consensus_queue.cc:260] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e594bb08ed9b4e009f02a139967b54a1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43663 } }
I20260927 21:33:37.707787 9335 raft_consensus.cc:388] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:37.707876 9335 raft_consensus.cc:482] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:37.707988 9335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:37.709867 9335 raft_consensus.cc:504] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e594bb08ed9b4e009f02a139967b54a1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43663 } }
I20260927 21:33:37.710233 9335 leader_election.cc:302] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e594bb08ed9b4e009f02a139967b54a1; no voters:
I20260927 21:33:37.710721 9335 leader_election.cc:288] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:37.710858 9340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:37.711886 9340 raft_consensus.cc:686] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [term 1 LEADER]: Becoming Leader. State: Replica: e594bb08ed9b4e009f02a139967b54a1, State: Running, Role: LEADER
I20260927 21:33:37.712460 9340 consensus_queue.cc:237] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e594bb08ed9b4e009f02a139967b54a1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43663 } }
I20260927 21:33:37.712613 9335 sys_catalog.cc:564] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:37.714989 9342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e594bb08ed9b4e009f02a139967b54a1. Latest consensus state: current_term: 1 leader_uuid: "e594bb08ed9b4e009f02a139967b54a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e594bb08ed9b4e009f02a139967b54a1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43663 } } }
I20260927 21:33:37.715342 9342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:37.716136 9349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:37.716029 9341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e594bb08ed9b4e009f02a139967b54a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e594bb08ed9b4e009f02a139967b54a1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43663 } } }
I20260927 21:33:37.716908 9341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:37.721251 9349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:37.728749 9349 catalog_manager.cc:1347] Generated new cluster ID: 7a23f1358fc24e61940ca855bbc10ba9
I20260927 21:33:37.728916 9349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:37.747767 9349 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:37.748904 9349 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:37.755664 9349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e594bb08ed9b4e009f02a139967b54a1: Generated new TSK 0
I20260927 21:33:37.756480 9349 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:33:37.812112 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:37.775637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43663 (local address 127.0.0.1:60590)
0927 21:33:37.776151 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:37.776170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:37.776318 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:37.776669 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:37.776675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:37.776686 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:33:37.777008 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:37.777019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.778068 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.778073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:37.779819 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:37.779833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.780486 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.780492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:37.780660 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:37.780706 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:33:37.811668 (+ 30962us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:33:37.811857 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.186.254:43663: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20260927 21:33:37.812104 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:37.775655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.3.186.254:43663)
0927 21:33:37.775850 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:37.775859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:37.776246 (+ 387us) server_negotiation.cc:407] Connection header received
0927 21:33:37.776459 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:37.776462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:37.776536 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:37.776634 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:33:37.777149 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.777929 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.779936 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:37.780339 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:37.780423 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:37.780896 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:33:37.811457 (+ 30561us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:33:37.811816 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60590: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 21:33:37.812283 9338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60590: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 21:33:37.813629 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9269
I20260927 21:33:37.814658 9330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:37.902832 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9269
2026-09-27T21:33:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (657 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (379 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (377 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:38 dist-test-slave-pqr8 krb5kdc[9379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:38 dist-test-slave-pqr8 krb5kdc[9379](info): set up 2 sockets
Sep 27 21:33:38 dist-test-slave-pqr8 krb5kdc[9379](info): commencing operation
krb5kdc: starting...
W20260927 21:33:39.706012 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:39 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:39.886735 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40941
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:40941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:40.182772 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:40.183048 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:40.183178 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:40.189795 9392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:40.189911 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:40.190007 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:40.190094 9392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:40.190153 9392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:40.199002 9392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:40941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:40.200160 9392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:40.201370 9392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:40.212127 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:40.212013 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:40.212311 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:40.213172 9392 server_base.cc:1034] running on GCE node
I20260927 21:33:40.213884 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:40.215032 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:40.216250 9392 hybrid_clock.cc:648] HybridClock initialized: now 1790544820216207 us; error 46 us; skew 500 ppm
I20260927 21:33:40.216553 9392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:40 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:40.219344 9392 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:33:40.220620 9392 webserver.cc:460] Webserver started at http://127.3.186.254:43077/ using document root <none> and password file <none>
I20260927 21:33:40.221251 9392 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:40.221383 9392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:40.221724 9392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:40.224385 9392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "bf937a35743a40379ff04de1985c8d43"
format_stamp: "Formatted at 2026-09-27 21:33:40 on dist-test-slave-pqr8"
server_key: "6018c7cf9a411bf5c089e5c5aad916fb"
server_key_iv: "d3f34af4a7adcb60b5808f788401489e"
server_key_version: "encryptionkey@0"
I20260927 21:33:40.225136 9392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "bf937a35743a40379ff04de1985c8d43"
format_stamp: "Formatted at 2026-09-27 21:33:40 on dist-test-slave-pqr8"
server_key: "6018c7cf9a411bf5c089e5c5aad916fb"
server_key_iv: "d3f34af4a7adcb60b5808f788401489e"
server_key_version: "encryptionkey@0"
I20260927 21:33:40.229869 9392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 21:33:40.232923 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:40.234099 9392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:40.234253 9392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "bf937a35743a40379ff04de1985c8d43"
format_stamp: "Formatted at 2026-09-27 21:33:40 on dist-test-slave-pqr8"
server_key: "6018c7cf9a411bf5c089e5c5aad916fb"
server_key_iv: "d3f34af4a7adcb60b5808f788401489e"
server_key_version: "encryptionkey@0"
I20260927 21:33:40.234419 9392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:40.255836 9392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:40.259558 9392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:40.259815 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:40.276578 9392 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:40941
I20260927 21:33:40.276646 9458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:40941 every 8 connection(s)
I20260927 21:33:40.277793 9392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:40.280495 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9392
I20260927 21:33:40.280714 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:40.281486 9459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:40.281574 3819 external_mini_cluster.cc:1428] Setting key 4a32ede5b06b31dfeaa3cfef80f33cd1
I20260927 21:33:40.293007 9459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43: Bootstrap starting.
Sep 27 21:33:40 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:40.295982 9459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:40.297070 9459 log.cc:824] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:40.299549 9459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43: No bootstrap required, opened a new log
I20260927 21:33:40.302160 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:40.283448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.3.186.254:40941)
0927 21:33:40.284150 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:40.284160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:40.284199 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:33:40.286107 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:40.286126 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:40.286477 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:40.286832 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:40.288196 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:40.289391 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:40.290598 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:40.291252 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:40.291465 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:40.295390 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:40.295396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:40.295408 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:40.295436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:40.298591 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:40.299213 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:40.299219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:40.299224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:40.299361 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:40.299758 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:40.299762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:40.299764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:40.300207 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:40.300495 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:40.300883 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 21:33:40.301239 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260927 21:33:40.307029 9459 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf937a35743a40379ff04de1985c8d43" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40941 } }
I20260927 21:33:40.307346 9459 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:40.307438 9459 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf937a35743a40379ff04de1985c8d43, State: Initialized, Role: FOLLOWER
I20260927 21:33:40.307972 9459 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf937a35743a40379ff04de1985c8d43" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40941 } }
I20260927 21:33:40.308202 9459 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:40.308358 9459 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:40.308530 9459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:40.310503 9459 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf937a35743a40379ff04de1985c8d43" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40941 } }
I20260927 21:33:40.310874 9459 leader_election.cc:302] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf937a35743a40379ff04de1985c8d43; no voters:
I20260927 21:33:40.311316 9459 leader_election.cc:288] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:40.311429 9464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:40.312047 9464 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [term 1 LEADER]: Becoming Leader. State: Replica: bf937a35743a40379ff04de1985c8d43, State: Running, Role: LEADER
I20260927 21:33:40.312568 9464 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf937a35743a40379ff04de1985c8d43" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40941 } }
I20260927 21:33:40.313048 9459 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:40.315405 9465 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf937a35743a40379ff04de1985c8d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf937a35743a40379ff04de1985c8d43" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40941 } } }
I20260927 21:33:40.315828 9465 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:40.316336 9466 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf937a35743a40379ff04de1985c8d43. Latest consensus state: current_term: 1 leader_uuid: "bf937a35743a40379ff04de1985c8d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf937a35743a40379ff04de1985c8d43" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40941 } } }
I20260927 21:33:40.316744 9466 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:40.320886 9472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:40.324793 9472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:40.332026 9472 catalog_manager.cc:1347] Generated new cluster ID: b6b756e9de194a08914c006d83500a27
I20260927 21:33:40.332121 9472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:40.348227 9472 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:40.349857 9472 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:40.365913 9472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf937a35743a40379ff04de1985c8d43: Generated new TSK 0
I20260927 21:33:40.366740 9472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:33:40.412890 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40941
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:40.712582 9485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:40.712809 9485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:40.712980 9485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:40.719888 9485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:40.719998 9485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:40.720160 9485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:40.729075 9485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:40.730300 9485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:40.731452 9485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:40.744122 9493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:40.744181 9491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:40.744204 9490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:40.744794 9485 server_base.cc:1034] running on GCE node
I20260927 21:33:40.745339 9485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:40.746227 9485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:40.747408 9485 hybrid_clock.cc:648] HybridClock initialized: now 1790544820747372 us; error 27 us; skew 500 ppm
I20260927 21:33:40.747679 9485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:40 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:40.750636 9485 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:33:40.751868 9485 webserver.cc:460] Webserver started at http://127.3.186.193:42899/ using document root <none> and password file <none>
I20260927 21:33:40.752473 9485 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:40.752610 9485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:40.752887 9485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:40.755618 9485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "90938a240af7426393ef93c4472117bf"
format_stamp: "Formatted at 2026-09-27 21:33:40 on dist-test-slave-pqr8"
server_key: "ca5b75bf26fba506fff48b932a366c9d"
server_key_iv: "2e28330bc9e781a8d38902b0aca20e87"
server_key_version: "encryptionkey@0"
I20260927 21:33:40.756369 9485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "90938a240af7426393ef93c4472117bf"
format_stamp: "Formatted at 2026-09-27 21:33:40 on dist-test-slave-pqr8"
server_key: "ca5b75bf26fba506fff48b932a366c9d"
server_key_iv: "2e28330bc9e781a8d38902b0aca20e87"
server_key_version: "encryptionkey@0"
I20260927 21:33:40.760917 9485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 21:33:40.764148 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:40.765240 9485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:40.765393 9485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "90938a240af7426393ef93c4472117bf"
format_stamp: "Formatted at 2026-09-27 21:33:40 on dist-test-slave-pqr8"
server_key: "ca5b75bf26fba506fff48b932a366c9d"
server_key_iv: "2e28330bc9e781a8d38902b0aca20e87"
server_key_version: "encryptionkey@0"
I20260927 21:33:40.765620 9485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:40.775373 9485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:40.778788 9485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:40.779048 9485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:40.779837 9485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:40.780990 9485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:40.781086 9485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:40.781219 9485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:40.781288 9485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:40.811255 9485 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:46823
I20260927 21:33:40.811326 9612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:46823 every 8 connection(s)
I20260927 21:33:40.812394 9485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:40.818379 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9485
I20260927 21:33:40.818569 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:40.819361 3819 external_mini_cluster.cc:1428] Setting key e0715f950cd18f2cd5dea1b9001c46b7
Sep 27 21:33:40 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:40.833384 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:40.814776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:42265 (local address 127.3.186.254:40941)
0927 21:33:40.815174 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:40.815178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:40.816092 (+ 914us) server_negotiation.cc:407] Connection header received
0927 21:33:40.817844 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:40.817849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:40.817913 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:40.818081 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:40.820072 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:40.821073 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:40.822661 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:40.823052 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:40.823189 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:40.826899 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:40.826907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:40.826914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:40.826954 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:40.830341 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:40.830979 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:40.830987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:40.830993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:40.831138 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:40.831503 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:40.831511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:40.831516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:40.831906 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:40.832069 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:40.832660 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 21:33:40.832907 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":199,"threads_started":1}
I20260927 21:33:40.833920 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:40.815316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40941 (local address 127.3.186.193:42265)
0927 21:33:40.815955 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:40.815989 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:40.817543 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:40.818350 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:40.818363 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:40.819083 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:40.819870 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:40.819894 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:40.821220 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:40.821225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:40.822514 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:40.822526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:40.823262 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:40.823270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:40.823521 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:40.824155 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:40.824175 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:40.826690 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:40.830471 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:40.830477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:40.830487 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:40.830858 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:40.831237 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:40.831240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:40.831242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:40.831394 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:40.832028 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:40.832051 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:40.832315 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 21:33:40.832514 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 21:33:40.832841 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":275,"threads_started":1}
I20260927 21:33:40.835245 9613 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40941
I20260927 21:33:40.835522 9613 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:40.836143 9613 heartbeater.cc:511] Master 127.3.186.254:40941 requested a full tablet report, sending...
I20260927 21:33:40.838188 9424 ts_manager.cc:194] Registered new tserver with Master: 90938a240af7426393ef93c4472117bf (127.3.186.193:46823)
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
I20260927 21:33:40.839669 9424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:42265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:33:40.862371 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40941
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:41.156889 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:41.157116 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:41.157196 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:41.163746 9619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:41.163828 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:41.163924 9619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:41.172546 9619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:41.173481 9619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:41.174607 9619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:41.186383 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:41.186426 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:41.186493 9627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:41.187083 9619 server_base.cc:1034] running on GCE node
I20260927 21:33:41.187605 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:41.188436 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:41.189597 9619 hybrid_clock.cc:648] HybridClock initialized: now 1790544821189550 us; error 41 us; skew 500 ppm
I20260927 21:33:41.189867 9619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.192619 9619 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:33:41.193935 9619 webserver.cc:460] Webserver started at http://127.3.186.194:36883/ using document root <none> and password file <none>
I20260927 21:33:41.194510 9619 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:41.194625 9619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:41.194890 9619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:41.197304 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "c7b8ced31e6a4227a80688efcd1a213a"
format_stamp: "Formatted at 2026-09-27 21:33:41 on dist-test-slave-pqr8"
server_key: "0255fd2c654bda7d171d10f2790af5c9"
server_key_iv: "1db4568be41232a5e35974f4edf15439"
server_key_version: "encryptionkey@0"
I20260927 21:33:41.198066 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "c7b8ced31e6a4227a80688efcd1a213a"
format_stamp: "Formatted at 2026-09-27 21:33:41 on dist-test-slave-pqr8"
server_key: "0255fd2c654bda7d171d10f2790af5c9"
server_key_iv: "1db4568be41232a5e35974f4edf15439"
server_key_version: "encryptionkey@0"
I20260927 21:33:41.202526 9619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:33:41.205386 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:41.206550 9619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:41.206696 9619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "c7b8ced31e6a4227a80688efcd1a213a"
format_stamp: "Formatted at 2026-09-27 21:33:41 on dist-test-slave-pqr8"
server_key: "0255fd2c654bda7d171d10f2790af5c9"
server_key_iv: "1db4568be41232a5e35974f4edf15439"
server_key_version: "encryptionkey@0"
I20260927 21:33:41.206857 9619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:41.241250 9619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:41.244701 9619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:41.244949 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:41.245747 9619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:41.246862 9619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:41.246935 9619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:41.247083 9619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:41.247130 9619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:41.282174 9619 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:45383
I20260927 21:33:41.282222 9746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:45383 every 8 connection(s)
I20260927 21:33:41.283267 9619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:41.287039 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9619
I20260927 21:33:41.287220 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:41.287645 3819 external_mini_cluster.cc:1428] Setting key 287fd7064f61f0573d373ad85320dfe3
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:41.303346 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.285569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:52741 (local address 127.3.186.254:40941)
0927 21:33:41.285742 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.285748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.286901 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 21:33:41.289290 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.289297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.289371 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.289475 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.291167 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.292249 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.294086 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.294814 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.294910 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.298208 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.298213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.298216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.298250 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.300692 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.301349 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.301358 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.301360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.301473 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.301908 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.301913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.301916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.302139 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.302284 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.302845 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.303039 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:33:41.304155 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.285992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40941 (local address 127.3.186.194:52741)
0927 21:33:41.286743 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:41.286791 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:41.289023 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:41.289722 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:41.289733 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:41.290374 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:41.290995 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:41.291015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.292397 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.292402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:41.293957 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:41.293968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.295021 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.295029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:41.295358 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.295982 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:41.296002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:41.298006 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:41.300833 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:41.300841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:41.300854 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:41.301232 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:41.301617 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:41.301622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:41.301625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:41.301797 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:41.302252 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:41.302259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:41.302530 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 21:33:41.302759 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 21:33:41.303206 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":289,"threads_started":1}
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:33:41.305405 9747 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40941
I20260927 21:33:41.305681 9747 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:41.306362 9747 heartbeater.cc:511] Master 127.3.186.254:40941 requested a full tablet report, sending...
I20260927 21:33:41.307637 9424 ts_manager.cc:194] Registered new tserver with Master: c7b8ced31e6a4227a80688efcd1a213a (127.3.186.194:45383)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
I20260927 21:33:41.308427 9424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:52741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:33:41.330127 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40941
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:41.621992 9752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:41.622219 9752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:41.622402 9752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:41.628782 9752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:41.628868 9752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:41.629041 9752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:41.637691 9752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:41.638691 9752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:41.639855 9752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:41.651654 9758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:41.651683 9757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:41.652112 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:41.652415 9752 server_base.cc:1034] running on GCE node
I20260927 21:33:41.652880 9752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:41.653765 9752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:41.654973 9752 hybrid_clock.cc:648] HybridClock initialized: now 1790544821654931 us; error 30 us; skew 500 ppm
I20260927 21:33:41.655254 9752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.658082 9752 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:33:41.659269 9752 webserver.cc:460] Webserver started at http://127.3.186.195:33087/ using document root <none> and password file <none>
I20260927 21:33:41.659833 9752 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:41.659924 9752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:41.660140 9752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:41.662626 9752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "503bea0d0d864e778c4bd0d7420dd28e"
format_stamp: "Formatted at 2026-09-27 21:33:41 on dist-test-slave-pqr8"
server_key: "06e8e563f7b884ab2ce35697de8d34ba"
server_key_iv: "60c516e6d2ebbbd8dc01af470a18e745"
server_key_version: "encryptionkey@0"
I20260927 21:33:41.663267 9752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "503bea0d0d864e778c4bd0d7420dd28e"
format_stamp: "Formatted at 2026-09-27 21:33:41 on dist-test-slave-pqr8"
server_key: "06e8e563f7b884ab2ce35697de8d34ba"
server_key_iv: "60c516e6d2ebbbd8dc01af470a18e745"
server_key_version: "encryptionkey@0"
I20260927 21:33:41.667626 9752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:33:41.670794 9767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:41.671890 9752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:41.672058 9752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "503bea0d0d864e778c4bd0d7420dd28e"
format_stamp: "Formatted at 2026-09-27 21:33:41 on dist-test-slave-pqr8"
server_key: "06e8e563f7b884ab2ce35697de8d34ba"
server_key_iv: "60c516e6d2ebbbd8dc01af470a18e745"
server_key_version: "encryptionkey@0"
I20260927 21:33:41.672262 9752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:41.683112 9752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:41.686378 9752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:41.686631 9752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:41.687431 9752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:41.688567 9752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:41.688668 9752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:41.688796 9752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:41.688866 9752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:41.719234 9752 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:38631
I20260927 21:33:41.719314 9879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:38631 every 8 connection(s)
I20260927 21:33:41.720340 9752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:41.724555 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9752
I20260927 21:33:41.724712 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:41.725076 3819 external_mini_cluster.cc:1428] Setting key 2cc2cf49dd92ae8106c97cbdf4a71e90
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:33:41.737758 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.722450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:54183 (local address 127.3.186.254:40941)
0927 21:33:41.722633 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.722640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.723635 (+ 995us) server_negotiation.cc:407] Connection header received
0927 21:33:41.725307 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.725313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.725406 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.725538 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.727277 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.728188 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.729632 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.729989 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.730105 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.733168 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.733173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.733175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.733201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.735470 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.736044 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.736047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.736049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.736172 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.736486 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.736489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.736491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.736717 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.736811 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.737301 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.737483 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:33:41.738660 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.722868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40941 (local address 127.3.186.195:54183)
0927 21:33:41.723505 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:41.723537 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:41.725052 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:41.725775 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:41.725784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:41.726424 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:41.727089 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:41.727134 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.728309 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.728312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:41.729447 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:41.729457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.730191 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.730196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:41.730363 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.730951 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:41.730970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:41.732988 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:41.735590 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:41.735595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:41.735604 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:41.735951 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:41.736252 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:41.736255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:41.736257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:41.736406 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:41.736803 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:41.736807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:41.737029 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 21:33:41.737240 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 21:33:41.737571 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":280,"threads_started":1}
I20260927 21:33:41.739899 9880 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40941
I20260927 21:33:41.740170 9880 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:41.740765 9880 heartbeater.cc:511] Master 127.3.186.254:40941 requested a full tablet report, sending...
I20260927 21:33:41.741961 9424 ts_manager.cc:194] Registered new tserver with Master: 503bea0d0d864e778c4bd0d7420dd28e (127.3.186.195:38631)
I20260927 21:33:41.742699 9424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:54183
I20260927 21:33:41.751493 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/test-admin.keytab.
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.775875 3819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 21:33:41.780668 9887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:41.778313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.3.186.193:46823)
0927 21:33:41.778877 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.778888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.778918 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:33:41.779123 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.779130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.779320 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.779466 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.780274 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50004: BlockingRecv error: recv got EOF from 127.0.0.1:50004 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
W20260927 21:33:41.780838 3819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.782821 3819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.784320 3819 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:41.799672 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.786438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.3.186.193:46823)
0927 21:33:41.786576 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.786584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.787117 (+ 533us) server_negotiation.cc:407] Connection header received
0927 21:33:41.787174 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.787179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.787242 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.787336 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.788614 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.789650 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.790824 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.791339 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.791439 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.794817 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.794827 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.794840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.794869 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.797351 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.798111 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.798117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.798119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.798232 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.798531 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.798534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.798536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.798839 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.799039 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.799262 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.799449 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 21:33:41.804648 9887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:41.803230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.3.186.193:46823)
0927 21:33:41.803420 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.803425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.803546 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:33:41.803688 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.803691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.803760 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.803846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.804443 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50012: BlockingRecv error: recv got EOF from 127.0.0.1:50012 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 21:33:41.804831 3819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.806208 3819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.807397 3819 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:41.821238 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.808444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.3.186.193:46823)
0927 21:33:41.808626 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.808631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.808812 (+ 181us) server_negotiation.cc:407] Connection header received
0927 21:33:41.808943 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.808946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.808994 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.809069 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.809983 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.810801 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.812029 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.812481 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.812610 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.816096 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.816101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.816104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.816130 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.818897 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.819509 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.819516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.819519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.819660 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.820009 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.820015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.820018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.820263 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.820514 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.820816 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.820994 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 21:33:41.824834 9887 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:41.823359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.3.186.193:46823)
0927 21:33:41.823547 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.823552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.823748 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:33:41.823880 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.823883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.823957 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.824027 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.824623 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50024: BlockingRecv error: recv got EOF from 127.0.0.1:50024 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 21:33:41.824954 3819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544773327888-3819-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.826349 3819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:41.827553 3819 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:33:41.840152 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.828662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.3.186.193:46823)
0927 21:33:41.828856 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.828861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.829002 (+ 141us) server_negotiation.cc:407] Connection header received
0927 21:33:41.829099 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.829103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.829148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.829221 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.830146 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.830818 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.831904 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.832293 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.832380 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.835407 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.835411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.835414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.835438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.837783 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.838375 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.838382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.838386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.838523 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.838860 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.838865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.838869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.839096 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.839384 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.839656 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.839832 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:33:41.841675 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9485
I20260927 21:33:41.842512 9613 heartbeater.cc:503] Master 127.3.186.254:40941 was elected leader, sending a full tablet report...
I20260927 21:33:41.852628 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.842264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.3.186.193:46823)
0927 21:33:41.842385 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.842391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.842729 (+ 338us) server_negotiation.cc:407] Connection header received
0927 21:33:41.842851 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.842854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.842905 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.842996 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.843817 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.844496 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.845678 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.846079 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.846207 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.848229 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.848235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.848240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.848270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.850452 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.851008 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.851013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.851017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.851130 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.851455 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.851460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.851463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.851693 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.851936 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.852204 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.852373 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:33:41.853147 9608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:41.932170 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9485
I20260927 21:33:41.948319 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9619
Sep 27 21:33:41 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:33:41.962965 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:41.949087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.3.186.194:45383)
0927 21:33:41.949673 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:41.949680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:41.949705 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:33:41.949761 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:41.949768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:41.949963 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:41.950089 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:41.950981 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.951969 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.953141 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:41.953936 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:41.954029 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:41.957650 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:41.957660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:41.957673 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:41.957711 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:41.960572 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.961157 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.961164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.961167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.961295 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:41.961716 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:41.961719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:41.961721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:41.962009 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:41.962207 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:41.962500 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:33:41.962685 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
I20260927 21:33:41.964126 9742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:42.042400 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9619
I20260927 21:33:42.058593 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9752
Sep 27 21:33:42 dist-test-slave-pqr8 krb5kdc[9379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:33:42.077735 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:42.059541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.3.186.195:38631)
0927 21:33:42.060130 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:42.060136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:42.060164 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:33:42.060222 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:42.060228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:42.060420 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:42.060541 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:42.061401 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:42.062409 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:42.063533 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:42.064039 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:42.064131 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:42.072344 (+ 8213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:42.072354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:42.072366 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:42.072408 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:42.075204 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:42.075816 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:42.075821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:42.075823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:42.075930 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:42.076393 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:42.076396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:42.076398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:42.076725 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:42.076940 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:42.077166 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 21:33:42.077383 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":311,"threads_started":1}
I20260927 21:33:42.078892 9875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:42.156530 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9752
I20260927 21:33:42.171478 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9392
I20260927 21:33:42.172520 9454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:42.270288 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9392
2026-09-27T21:33:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3616 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:42 dist-test-slave-pqr8 krb5kdc[9907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:42 dist-test-slave-pqr8 krb5kdc[9907](info): set up 2 sockets
Sep 27 21:33:42 dist-test-slave-pqr8 krb5kdc[9907](info): commencing operation
krb5kdc: starting...
W20260927 21:33:43.326081 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:43 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab.
Entry for principal oryx/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab.
I20260927 21:33:43.498340 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44169
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab
--principal=oryx/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:43.795184 9920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:43.795475 9920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:43.795627 9920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:43.802557 9920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:43.802675 9920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:43.802765 9920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:43.802876 9920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:43.802948 9920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:43.812081 9920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab
--principal=oryx/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:43.813136 9920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:43.814357 9920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:43.825098 9926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:43.825181 9925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:43.825343 9928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:43.826074 9920 server_base.cc:1034] running on GCE node
I20260927 21:33:43.826871 9920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:43.828193 9920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:43.829430 9920 hybrid_clock.cc:648] HybridClock initialized: now 1790544823829374 us; error 52 us; skew 500 ppm
I20260927 21:33:43.829779 9920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:43 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:43.832651 9920 init.cc:377] Logged in from keytab as oryx/127.3.186.254@KRBTEST.COM (short username oryx)
I20260927 21:33:43.834004 9920 webserver.cc:460] Webserver started at http://127.3.186.254:35291/ using document root <none> and password file <none>
I20260927 21:33:43.834602 9920 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:43.834736 9920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:43.835006 9920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:43.837709 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "d1ae9d85f2c54051a7c7a95f68009e8b"
format_stamp: "Formatted at 2026-09-27 21:33:43 on dist-test-slave-pqr8"
server_key: "683d4ad3ebcf0aeb0fe30898b212f640"
server_key_iv: "ff131037a988b56c17de2a11953259ec"
server_key_version: "encryptionkey@0"
I20260927 21:33:43.838423 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "d1ae9d85f2c54051a7c7a95f68009e8b"
format_stamp: "Formatted at 2026-09-27 21:33:43 on dist-test-slave-pqr8"
server_key: "683d4ad3ebcf0aeb0fe30898b212f640"
server_key_iv: "ff131037a988b56c17de2a11953259ec"
server_key_version: "encryptionkey@0"
I20260927 21:33:43.842990 9920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 21:33:43.845976 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:43.847188 9920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:43.847378 9920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "d1ae9d85f2c54051a7c7a95f68009e8b"
format_stamp: "Formatted at 2026-09-27 21:33:43 on dist-test-slave-pqr8"
server_key: "683d4ad3ebcf0aeb0fe30898b212f640"
server_key_iv: "ff131037a988b56c17de2a11953259ec"
server_key_version: "encryptionkey@0"
I20260927 21:33:43.847563 9920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:43.866804 9920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:43.870556 9920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:43.870808 9920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:43.887384 9920 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44169
I20260927 21:33:43.887436 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44169 every 8 connection(s)
I20260927 21:33:43.888558 9920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:43.891795 9987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:43.892298 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 9920
I20260927 21:33:43.892468 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:43.893065 3819 external_mini_cluster.cc:1428] Setting key 421760f9c1e520c125c922b29838dc6a
I20260927 21:33:43.904408 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Bootstrap starting.
Sep 27 21:33:43 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:43.907581 9987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:43.908592 9987 log.cc:824] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:43.911098 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: No bootstrap required, opened a new log
I20260927 21:33:43.913479 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:43.895004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.3.186.254:44169)
0927 21:33:43.895714 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:43.895723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:43.895761 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:33:43.897248 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:43.897266 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:43.897636 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:43.897953 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:43.899557 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:43.900727 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:43.902424 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:43.902997 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:43.903111 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:43.907212 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:43.907219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:43.907231 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:43.907259 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:43.910039 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:43.910643 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:43.910655 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:43.910660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:43.910806 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:43.911136 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:43.911142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:43.911145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:43.911607 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:43.911907 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:43.912271 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 21:33:43.912636 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
I20260927 21:33:43.918551 9987 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } }
I20260927 21:33:43.918867 9987 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:43.918968 9987 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ae9d85f2c54051a7c7a95f68009e8b, State: Initialized, Role: FOLLOWER
I20260927 21:33:43.919505 9987 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } }
I20260927 21:33:43.919727 9987 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:43.919865 9987 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:43.920037 9987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:43.922070 9987 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } }
I20260927 21:33:43.922500 9987 leader_election.cc:302] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1ae9d85f2c54051a7c7a95f68009e8b; no voters:
I20260927 21:33:43.923066 9987 leader_election.cc:288] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:43.923197 9992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:43.923841 9992 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 LEADER]: Becoming Leader. State: Replica: d1ae9d85f2c54051a7c7a95f68009e8b, State: Running, Role: LEADER
I20260927 21:33:43.924391 9992 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } }
I20260927 21:33:43.925040 9987 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:43.927456 9994 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ae9d85f2c54051a7c7a95f68009e8b. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } }
I20260927 21:33:43.927433 9993 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } }
I20260927 21:33:43.927800 9994 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:43.927809 9993 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:43.929325 10000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:43.934136 10000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:43.941866 10000 catalog_manager.cc:1347] Generated new cluster ID: 795d1935d3b44545a5685909570ec94f
I20260927 21:33:43.942109 10000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:43.983053 10000 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:43.984673 10000 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:43.998790 10000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Generated new TSK 0
I20260927 21:33:43.999658 10000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab.
Entry for principal oryx/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab.
I20260927 21:33:44.046329 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44169
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:44.363363 10013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:44.363583 10013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:44.363719 10013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:44.370167 10013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:44.370272 10013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:44.370435 10013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:44.379267 10013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:44.380299 10013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:44.381443 10013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:44.393384 10019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:44.393352 10021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:44.393787 10018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:44.394227 10013 server_base.cc:1034] running on GCE node
I20260927 21:33:44.394722 10013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:44.395538 10013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:44.396757 10013 hybrid_clock.cc:648] HybridClock initialized: now 1790544824396698 us; error 59 us; skew 500 ppm
I20260927 21:33:44.397059 10013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:44 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544824, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:44.399940 10013 init.cc:377] Logged in from keytab as oryx/127.3.186.193@KRBTEST.COM (short username oryx)
I20260927 21:33:44.401250 10013 webserver.cc:460] Webserver started at http://127.3.186.193:45415/ using document root <none> and password file <none>
I20260927 21:33:44.401889 10013 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:44.402014 10013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:44.402271 10013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:44.404771 10013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "3c8b773ec5984807a386ae0394a7472d"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "b7c5d5e81b4a9b28889f09108347776b"
server_key_iv: "203eccabe9d80fe20b7a238f5f6f40ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:44.405445 10013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "3c8b773ec5984807a386ae0394a7472d"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "b7c5d5e81b4a9b28889f09108347776b"
server_key_iv: "203eccabe9d80fe20b7a238f5f6f40ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:44.410018 10013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 21:33:44.412963 10028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:44.414139 10013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 21:33:44.414258 10013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "3c8b773ec5984807a386ae0394a7472d"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "b7c5d5e81b4a9b28889f09108347776b"
server_key_iv: "203eccabe9d80fe20b7a238f5f6f40ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:44.414400 10013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:44.435452 10013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:44.438866 10013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:44.439090 10013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:44.439841 10013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:44.440919 10013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:44.440982 10013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:44.441061 10013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:44.441102 10013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:44.470721 10013 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:46147
I20260927 21:33:44.470790 10140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:46147 every 8 connection(s)
I20260927 21:33:44.471802 10013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:44.472946 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10013
I20260927 21:33:44.473131 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:44.473910 3819 external_mini_cluster.cc:1428] Setting key 9defffc23160b102a2b5233aa96d5d41
Sep 27 21:33:44 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544824, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:44.492362 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:44.474258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:39209 (local address 127.3.186.254:44169)
0927 21:33:44.474686 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:44.474690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:44.475614 (+ 924us) server_negotiation.cc:407] Connection header received
0927 21:33:44.477260 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:44.477268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:44.477357 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:44.477468 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:44.479200 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.480383 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.481897 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.482568 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.482704 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:44.485885 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:44.485893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:44.485901 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:44.485942 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:44.489414 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:44.490092 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:44.490099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:44.490104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:44.490244 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:44.490618 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:44.490626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:44.490632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:44.491017 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:44.491172 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:44.491835 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 21:33:44.492024 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260927 21:33:44.493125 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:44.474752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:39209)
0927 21:33:44.475477 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:44.475515 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:44.477025 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:44.477738 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:44.477751 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:44.478494 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:44.479046 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:44.479063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.480533 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.480537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:44.481765 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:44.481776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.482741 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.482745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:44.482922 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:44.483497 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:44.483516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:44.485693 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:44.489581 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:44.489599 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:44.489608 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:44.489971 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:44.490348 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:44.490351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:44.490353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:44.490507 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:44.491133 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:44.491138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:44.491406 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 21:33:44.491735 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 21:33:44.492136 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":304,"threads_started":1}
WARNING: no policy specified for oryx/127.3.186.194@KRBTEST.COM; defaulting to no policy
I20260927 21:33:44.494597 10141 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44169
I20260927 21:33:44.494890 10141 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:44.495524 10141 heartbeater.cc:511] Master 127.3.186.254:44169 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.194@KRBTEST.COM" created.
I20260927 21:33:44.497376 9952 ts_manager.cc:194] Registered new tserver with Master: 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147)
I20260927 21:33:44.498884 9952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:39209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab.
Entry for principal oryx/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab.
I20260927 21:33:44.516660 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44169
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:44.810580 10147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:44.810802 10147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:44.810976 10147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:44.817482 10147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:44.817634 10147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:44.817838 10147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:44.826622 10147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:44.827615 10147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:44.828707 10147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:44.840716 10155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:44.840788 10153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:44.840780 10152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:44.841301 10147 server_base.cc:1034] running on GCE node
I20260927 21:33:44.841964 10147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:44.842804 10147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:44.844014 10147 hybrid_clock.cc:648] HybridClock initialized: now 1790544824843969 us; error 39 us; skew 500 ppm
I20260927 21:33:44.844277 10147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:44 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544824, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:44.847235 10147 init.cc:377] Logged in from keytab as oryx/127.3.186.194@KRBTEST.COM (short username oryx)
I20260927 21:33:44.848503 10147 webserver.cc:460] Webserver started at http://127.3.186.194:35711/ using document root <none> and password file <none>
I20260927 21:33:44.849113 10147 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:44.849243 10147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:44.849555 10147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:44.852293 10147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "c9c8e6fc0592439789a1a76296c757c2"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "aa24ad900bbab055f9f731587ae68f99"
server_key_iv: "e6a9cae2cdbcbb8372461747a9f2c738"
server_key_version: "encryptionkey@0"
I20260927 21:33:44.852970 10147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "c9c8e6fc0592439789a1a76296c757c2"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "aa24ad900bbab055f9f731587ae68f99"
server_key_iv: "e6a9cae2cdbcbb8372461747a9f2c738"
server_key_version: "encryptionkey@0"
I20260927 21:33:44.857620 10147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260927 21:33:44.860680 10162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:44.861989 10147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:33:44.862138 10147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "c9c8e6fc0592439789a1a76296c757c2"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "aa24ad900bbab055f9f731587ae68f99"
server_key_iv: "e6a9cae2cdbcbb8372461747a9f2c738"
server_key_version: "encryptionkey@0"
I20260927 21:33:44.862310 10147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:44.881935 10147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:44.885573 10147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:44.885807 10147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:44.886650 10147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:44.888144 10147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:44.888242 10147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:44.888353 10147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:44.888432 10147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:44.920221 10147 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:34727
I20260927 21:33:44.920290 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:34727 every 8 connection(s)
I20260927 21:33:44.921424 10147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:44.931137 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10147
I20260927 21:33:44.931313 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:44.931928 3819 external_mini_cluster.cc:1428] Setting key 800e87ba21909a7fd3dd1b7250cca5b3
Sep 27 21:33:44 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544824, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:44.939702 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:44.923657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:44967 (local address 127.3.186.254:44169)
0927 21:33:44.923838 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:44.923842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:44.924801 (+ 959us) server_negotiation.cc:407] Connection header received
0927 21:33:44.926552 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:44.926556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:44.926617 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:44.926690 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:44.927919 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.928634 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.930283 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.930646 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.930732 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:44.934437 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:44.934442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:44.934445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:44.934474 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:44.936935 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:44.937680 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:44.937690 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:44.937692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:44.937803 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:44.938201 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:44.938205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:44.938208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:44.938455 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:44.938599 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:44.939185 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 21:33:44.939386 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:33:44.940864 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:44.924037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:44967)
0927 21:33:44.924650 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:44.924688 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:44.926318 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:44.926872 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:44.926879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:44.927305 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:44.927781 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:44.927796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.928767 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.928772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:44.930175 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:44.930187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:44.930853 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:44.930862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:44.931138 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:44.931740 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:44.931768 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:44.934229 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:44.937065 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:44.937074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:44.937087 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:44.937474 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:44.937924 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:44.937929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:44.937933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:44.938094 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:44.938582 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:44.938588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:44.938863 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 21:33:44.939090 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 21:33:44.939538 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":233,"threads_started":1}
I20260927 21:33:44.942117 10275 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44169
I20260927 21:33:44.942389 10275 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:44.942989 10275 heartbeater.cc:511] Master 127.3.186.254:44169 requested a full tablet report, sending...
I20260927 21:33:44.944173 9952 ts_manager.cc:194] Registered new tserver with Master: c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727)
I20260927 21:33:44.944901 9952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:44967
WARNING: no policy specified for oryx/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab.
Entry for principal oryx/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab.
I20260927 21:33:44.974021 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44169
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:45.271111 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:45.271368 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:45.271515 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:45.278378 10280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:45.278520 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:45.278677 10280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:45.287657 10280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:45.288707 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:45.289932 10280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:45.301708 10286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:45.301731 10285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:45.301797 10288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:45.302680 10280 server_base.cc:1034] running on GCE node
I20260927 21:33:45.303157 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:45.303918 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:45.305109 10280 hybrid_clock.cc:648] HybridClock initialized: now 1790544825305065 us; error 32 us; skew 500 ppm
I20260927 21:33:45.305400 10280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544825, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:45.308264 10280 init.cc:377] Logged in from keytab as oryx/127.3.186.195@KRBTEST.COM (short username oryx)
I20260927 21:33:45.309608 10280 webserver.cc:460] Webserver started at http://127.3.186.195:42237/ using document root <none> and password file <none>
I20260927 21:33:45.310285 10280 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:45.310401 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:45.310686 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:45.313195 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "7d833d302caa47a387a8ae46f66f8824"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "676d94166b390da2e1c1cf5c44a9ea18"
server_key_iv: "bedcf9002ed5ed52c397f15e5b1d7406"
server_key_version: "encryptionkey@0"
I20260927 21:33:45.313905 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "7d833d302caa47a387a8ae46f66f8824"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "676d94166b390da2e1c1cf5c44a9ea18"
server_key_iv: "bedcf9002ed5ed52c397f15e5b1d7406"
server_key_version: "encryptionkey@0"
I20260927 21:33:45.318543 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:45.321601 10295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:45.322729 10280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:45.322881 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "7d833d302caa47a387a8ae46f66f8824"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "676d94166b390da2e1c1cf5c44a9ea18"
server_key_iv: "bedcf9002ed5ed52c397f15e5b1d7406"
server_key_version: "encryptionkey@0"
I20260927 21:33:45.323055 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:45.336504 10280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:45.339967 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:45.340216 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:45.340992 10280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:45.342384 10280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:45.342490 10280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:45.342623 10280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:45.342713 10280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:45.377277 10280 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:37289
I20260927 21:33:45.377313 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:37289 every 8 connection(s)
I20260927 21:33:45.378513 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:45.388650 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10280
I20260927 21:33:45.388810 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:45.389307 3819 external_mini_cluster.cc:1428] Setting key 4d47be3c41132788cbebe5766e83c032
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544825, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:45.396759 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.380992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:49781 (local address 127.3.186.254:44169)
0927 21:33:45.381156 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.381161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.382275 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 21:33:45.383897 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.383904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.383996 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.384098 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.385399 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.386165 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.387650 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.388063 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.388186 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.391977 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.391982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.391987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.392020 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.394461 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.395104 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.395107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.395109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.395209 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.395534 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.395537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.395539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.395763 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.395865 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.396391 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.396541 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:33:45.397652 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.381443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:49781)
0927 21:33:45.382126 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:45.382164 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:45.383619 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:45.384307 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:45.384314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:45.384754 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:45.385242 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.385257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.386303 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.386312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.387511 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.387521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.388268 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.388277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.388519 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.389166 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:45.389185 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:45.391774 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:45.394582 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.394588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.394598 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.394983 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.395290 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.395293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.395295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.395450 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.395855 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:45.395860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:45.396127 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 21:33:45.396332 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 21:33:45.396648 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":244,"threads_started":1}
I20260927 21:33:45.398765 10408 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44169
I20260927 21:33:45.399029 10408 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:45.399664 10408 heartbeater.cc:511] Master 127.3.186.254:44169 requested a full tablet report, sending...
I20260927 21:33:45.400938 9952 ts_manager.cc:194] Registered new tserver with Master: 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289)
I20260927 21:33:45.401734 9952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:49781
I20260927 21:33:45.403995 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.186.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.253.keytab.
Entry for principal oryx/127.3.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.253.keytab.
I20260927 21:33:45.446769 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:42241
--webserver_interface=127.3.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.253.keytab
--principal=oryx/127.3.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44169,127.3.186.253:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:33:45.501286 10141 heartbeater.cc:503] Master 127.3.186.254:44169 was elected leader, sending a full tablet report...
W20260927 21:33:45.746801 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:45.747030 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:45.747177 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:45.753929 10413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:45.754027 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:45.754119 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:45.754199 10413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:45.754264 10413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:45.762830 10413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44169,127.3.186.253:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.253.keytab
--principal=oryx/127.3.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
W20260927 21:33:45.763114 10413 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.186.254:44169,127.3.186.253:42241'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 21:33:45.763935 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:45.765093 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:45.775730 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:45.775863 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:45.775873 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:45.776396 10413 server_base.cc:1034] running on GCE node
I20260927 21:33:45.776980 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:45.778187 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:45.779381 10413 hybrid_clock.cc:648] HybridClock initialized: now 1790544825779316 us; error 47 us; skew 500 ppm
I20260927 21:33:45.779628 10413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544825, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:45.782784 10413 init.cc:377] Logged in from keytab as oryx/127.3.186.253@KRBTEST.COM (short username oryx)
I20260927 21:33:45.786633 10413 webserver.cc:460] Webserver started at http://127.3.186.253:41921/ using document root <none> and password file <none>
I20260927 21:33:45.787454 10413 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:45.787562 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:45.787832 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:45.790835 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/instance:
uuid: "8f8760c6f12d4916850047930a9c2e3d"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "499d3bad4e8141e74e23e0c320a4e803"
server_key_iv: "fe6db8a2bdcc268821d2f454fe49d573"
server_key_version: "encryptionkey@0"
I20260927 21:33:45.791563 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal/instance:
uuid: "8f8760c6f12d4916850047930a9c2e3d"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "499d3bad4e8141e74e23e0c320a4e803"
server_key_iv: "fe6db8a2bdcc268821d2f454fe49d573"
server_key_version: "encryptionkey@0"
I20260927 21:33:45.795971 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:33:45.798990 10428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:45.800139 10413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:45.800297 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
uuid: "8f8760c6f12d4916850047930a9c2e3d"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "499d3bad4e8141e74e23e0c320a4e803"
server_key_iv: "fe6db8a2bdcc268821d2f454fe49d573"
server_key_version: "encryptionkey@0"
I20260927 21:33:45.800489 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:45.811312 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:45.815049 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:45.815275 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:45.831796 10413 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.253:42241
I20260927 21:33:45.831815 10479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.253:42241 every 8 connection(s)
I20260927 21:33:45.832959 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
I20260927 21:33:45.836180 10480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:45.841684 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10413
I20260927 21:33:45.842360 10480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544825, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.253@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
I20260927 21:33:45.859798 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.845045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.3.186.254:44169)
0927 21:33:45.845218 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.845226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.845784 (+ 558us) server_negotiation.cc:407] Connection header received
0927 21:33:45.846726 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.846733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.846822 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.846921 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.848064 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.848838 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.850526 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.851078 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.851166 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.854208 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.854212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.854216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.854242 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.856946 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.857687 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.857693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.857696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.857819 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.858241 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.858246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.858249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.858534 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.858680 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.859278 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.859470 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:33:45.860685 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.845003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:51544)
0927 21:33:45.845656 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:45.845687 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:45.846490 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:45.847003 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:45.847009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:45.847587 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:45.847896 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.847910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.848975 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.848981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.850377 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.850392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.851258 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.851265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.851494 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.852144 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:45.852167 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:45.854037 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:45.857067 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.857080 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.857090 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.857566 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.857927 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.857932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.857936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.858127 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.858656 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:45.858663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:45.858938 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 21:33:45.859191 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 21:33:45.859611 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
I20260927 21:33:45.860700 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.843677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.3.186.253:42241)
0927 21:33:45.844294 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.844303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.844336 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:33:45.846480 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.846498 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.846715 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.846864 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.848160 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.849573 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.850869 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.851555 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.851716 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.855502 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.855510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.855519 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.855547 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.858036 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.858643 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.858655 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.858657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.858769 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.859130 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.859137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.859141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.859440 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.859721 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.860245 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.860427 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":286,"threads_started":1}
I20260927 21:33:45.862353 10480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544825, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.253@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
I20260927 21:33:45.872871 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.863215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55712)
0927 21:33:45.863461 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:45.863474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:45.863548 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:45.863762 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:45.863765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:45.864036 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:45.864145 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.864151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.865017 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.865020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.865879 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.865889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.866544 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.866548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.866682 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.867099 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:45.867113 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:45.868785 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:45.871209 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.871212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.871215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.871544 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.871804 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.871807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.871809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.871912 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.872293 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:45.872295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:45.872412 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 21:33:45.872508 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:33:45.872667 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 21:33:45.873010 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.863260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.3.186.253:42241)
0927 21:33:45.863417 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.863423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.863535 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:33:45.863617 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.863620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.863665 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.863734 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.864246 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.864888 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.865982 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.866430 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.866535 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.868941 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.868945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.868948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.868973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.871104 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.871632 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.871635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.871637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.871724 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.871994 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.871996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.871998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.872213 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.872301 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.872573 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.872764 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":61}
I20260927 21:33:45.876953 10480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Bootstrap starting.
I20260927 21:33:45.879652 10480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:45.880546 10480 log.cc:824] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:45.882889 10480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: No bootstrap required, opened a new log
I20260927 21:33:45.889503 10480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } }
I20260927 21:33:45.889830 10480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:45.889914 10480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8760c6f12d4916850047930a9c2e3d, State: Initialized, Role: FOLLOWER
I20260927 21:33:45.890432 10480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } }
I20260927 21:33:45.891170 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } } }
I20260927 21:33:45.891474 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:33:45.891993 10480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:33:45.899571 10497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:33:45.899693 10497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:33:45.901793 10413 master_runner.cc:428] Detected that this master 8f8760c6f12d4916850047930a9c2e3d is joining an existing cluster
I20260927 21:33:45.901877 10413 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.186.253:42241
I20260927 21:33:45.921619 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.905312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.3.186.254:44169)
0927 21:33:45.905468 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.905475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.906282 (+ 807us) server_negotiation.cc:407] Connection header received
0927 21:33:45.906507 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.906512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.906589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.906681 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.907599 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.908283 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.909724 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.910161 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.910288 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.911908 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.911914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.911919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.911951 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.915308 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.919257 (+ 3949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.919266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.919272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.919427 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.919885 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.919892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.919894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.920205 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.920320 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.921046 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.921267 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:33:45.922573 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.905395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:51548)
0927 21:33:45.906183 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:45.906205 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:45.906340 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:45.906699 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:45.906703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:45.907180 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:45.907472 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.907480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.908450 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.908457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.909572 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.909588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.910326 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.910333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.910503 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.911023 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:45.911038 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:45.911748 (+ 710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:45.915419 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.915422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.915425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.919114 (+ 3689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.919576 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.919580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.919584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.919755 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.920399 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:45.920403 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:45.920644 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 21:33:45.921824 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0927 21:33:45.922138 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":536,"threads_started":1}
I20260927 21:33:45.937024 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.905943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.3.186.253:42241)
0927 21:33:45.906124 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.906130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.912936 (+ 6806us) server_negotiation.cc:407] Connection header received
0927 21:33:45.913072 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.913079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.913148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.913245 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.919018 (+ 5773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.920089 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.922124 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.922863 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.922947 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.925058 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.925064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.925070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.925110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.929042 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.934395 (+ 5353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.934403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.934409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.934557 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.934992 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.935000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.935005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.935559 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.935709 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.936228 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.936482 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:33:45.938249 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.906272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55718)
0927 21:33:45.912835 (+ 6563us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:45.912851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:45.912968 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:45.913273 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:45.913279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:45.914809 (+ 1530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:45.918855 (+ 4046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.918867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.920361 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.920366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.921958 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.921971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.923214 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.923222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.923392 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.923924 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:45.923943 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:45.924890 (+ 947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:45.933623 (+ 8733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.933631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.933637 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.934264 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.934706 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.934713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.934718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.934878 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.935781 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:45.935788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:45.935940 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 21:33:45.937333 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0927 21:33:45.937669 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6451,"thread_start_us":6777,"threads_started":1}
WARNING: no policy specified for oryx/127.3.186.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.252@KRBTEST.COM" created.
Sep 27 21:33:45 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.254@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
I20260927 21:33:45.947587 10275 heartbeater.cc:503] Master 127.3.186.254:44169 was elected leader, sending a full tablet report...
I20260927 21:33:45.954003 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.928468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.3.186.253:42241)
0927 21:33:45.937292 (+ 8824us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:45.937300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:45.939605 (+ 2305us) server_negotiation.cc:407] Connection header received
0927 21:33:45.939933 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:45.939938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:45.940017 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:45.940112 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:45.941417 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.942450 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.944181 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.944926 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.945057 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.948695 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:45.948700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:45.948703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:45.948727 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:45.951471 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.952106 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.952111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.952116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.952265 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:45.952681 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:45.952686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:45.952690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:45.952918 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:45.953063 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:45.953436 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 21:33:45.953666 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8736,"thread_start_us":10459,"threads_started":1}
I20260927 21:33:45.954066 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:45.928184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55730)
0927 21:33:45.939498 (+ 11314us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:45.939536 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:45.939730 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:45.940179 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:45.940189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:45.940986 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:45.941250 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.941264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.942594 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.942602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.944056 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:45.944067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:45.945065 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:45.945069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:45.945186 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:45.945812 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:45.945831 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:45.948521 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:45.951579 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.951589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.951596 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.951992 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.952394 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:45.952398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:45.952401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:45.952580 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:45.953042 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:45.953050 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:45.953200 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 21:33:45.953359 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 21:33:45.953665 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11133,"thread_start_us":11019,"threads_started":1}
I20260927 21:33:45.955765 9952 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.186.253:42241: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d"
member_type: NON_VOTER
last_known_addr {
host: "127.3.186.253"
port: 42241
}
attrs {
promote: true
}
}
dest_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b"
cas_config_opid_index: -1
I20260927 21:33:45.956604 9952 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } }
I20260927 21:33:45.963860 10455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:45.968714 10455 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Refusing update from remote peer d1ae9d85f2c54051a7c7a95f68009e8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 21:33:45.969645 10511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.252.keytab.
Entry for principal oryx/127.3.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.252.keytab.
I20260927 21:33:45.973982 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:36899
--webserver_interface=127.3.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.252.keytab
--principal=oryx/127.3.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:33:45.975381 10510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:45.976909 10511 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.186.253:42241
I20260927 21:33:45.977440 10413 master.cc:537] Master@127.3.186.253:42241 shutting down...
I20260927 21:33:45.978853 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:45.979103 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:46.001430 10455 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.013644 10520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.014078 10520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:46.014122 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ae9d85f2c54051a7c7a95f68009e8b. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.014346 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:46.014492 10516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.014808 10516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:46.017135 10413 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 LEARNER]: Raft consensus shutting down.
I20260927 21:33:46.017592 10413 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 LEARNER]: Raft consensus is shut down!
I20260927 21:33:46.017755 10413 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: stopping tablet replica
I20260927 21:33:46.026012 10413 master.cc:552] Master@127.3.186.253:42241 shutdown complete.
I20260927 21:33:46.026100 10413 master_runner.cc:305] Clearing existing system tablet
I20260927 21:33:46.026614 10413 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:46.026736 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:46.029589 10413 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:46.033844 10522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:46.034231 10413 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:46.034569 10413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:46.034680 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
uuid: "8f8760c6f12d4916850047930a9c2e3d"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "499d3bad4e8141e74e23e0c320a4e803"
server_key_iv: "fe6db8a2bdcc268821d2f454fe49d573"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.039240 10413 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:46.045871 10413 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:33:46.046061 10413 log.cc:1196] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 21:33:46.046561 10413 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Deleting consensus metadata
I20260927 21:33:46.046864 10413 master_runner.cc:315] Copying system tablet from 127.3.186.254:44169
I20260927 21:33:46.049649 10413 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: tablet copy: Beginning tablet copy session from remote peer at address 127.3.186.254:44169
I20260927 21:33:46.065112 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.050783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.3.186.254:44169)
0927 21:33:46.050992 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.051000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.051026 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:33:46.051272 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.051278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.051400 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.051475 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.052674 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.053795 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.055645 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.056298 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.056442 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.058664 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.058671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.058676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.058714 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.062166 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.062923 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.062930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.062935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.063065 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.063510 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.063515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.063518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.063825 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.063982 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.064467 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.064730 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:33:46.065253 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.050435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:51554)
0927 21:33:46.050926 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.050945 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.051084 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.051704 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.051709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.052214 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.052487 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.052500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.053961 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.053969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.055489 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.055513 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.056476 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.056482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.056689 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.057373 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.057399 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.058435 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.062292 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.062297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.062301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.062805 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.063188 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.063195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.063202 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.063386 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.063969 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.063975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.064182 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.064369 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.064710 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":608,"threads_started":1}
I20260927 21:33:46.066260 9972 tablet_copy_service.cc:140] P d1ae9d85f2c54051a7c7a95f68009e8b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8f8760c6f12d4916850047930a9c2e3d ({username='oryx', principal='oryx/127.3.186.253@KRBTEST.COM'} at 127.0.0.1:51554)
I20260927 21:33:46.066540 9972 tablet_copy_service.cc:161] P d1ae9d85f2c54051a7c7a95f68009e8b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8f8760c6f12d4916850047930a9c2e3d at {username='oryx', principal='oryx/127.3.186.253@KRBTEST.COM'} at 127.0.0.1:51554: session id = 8f8760c6f12d4916850047930a9c2e3d-00000000000000000000000000000000
I20260927 21:33:46.070941 9972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:33:46.072770 10413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:33:46.079015 10413 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: tablet copy: Starting download of 0 data blocks...
I20260927 21:33:46.079285 10413 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: tablet copy: Starting download of 1 WAL segments...
I20260927 21:33:46.081745 10413 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:33:46.085615 9972 tablet_copy_service.cc:342] P d1ae9d85f2c54051a7c7a95f68009e8b: Request end of tablet copy session 8f8760c6f12d4916850047930a9c2e3d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.186.253@KRBTEST.COM'} at 127.0.0.1:51554
I20260927 21:33:46.085779 9972 tablet_copy_service.cc:433] P d1ae9d85f2c54051a7c7a95f68009e8b: ending tablet copy session 8f8760c6f12d4916850047930a9c2e3d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8f8760c6f12d4916850047930a9c2e3d
I20260927 21:33:46.097005 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:46.099843 10529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:46.100263 10530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:46.115056 10532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:46.115298 10413 server_base.cc:1034] running on GCE node
I20260927 21:33:46.115573 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:46.116225 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:46.117425 10413 hybrid_clock.cc:648] HybridClock initialized: now 1790544826117366 us; error 41 us; skew 500 ppm
I20260927 21:33:46.117645 10413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:46.119652 10413 init.cc:377] Logged in from keytab as oryx/127.3.186.253@KRBTEST.COM (short username oryx)
I20260927 21:33:46.125831 10413 webserver.cc:460] Webserver started at http://127.3.186.253:37171/ using document root <none> and password file <none>
I20260927 21:33:46.126173 10413 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:46.126256 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:46.129258 10413 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:46.131266 10539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:46.131633 10413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:33:46.131786 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
uuid: "8f8760c6f12d4916850047930a9c2e3d"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "499d3bad4e8141e74e23e0c320a4e803"
server_key_iv: "fe6db8a2bdcc268821d2f454fe49d573"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.132001 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:46.154635 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:46.158519 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:46.198490 10413 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.253:42241
I20260927 21:33:46.198544 10590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.253:42241 every 8 connection(s)
I20260927 21:33:46.199580 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
I20260927 21:33:46.205899 10591 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:33:46.210048 10591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Bootstrap starting.
I20260927 21:33:46.224496 10591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:33:46.224934 10591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: Bootstrap complete.
I20260927 21:33:46.226089 10591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } }
I20260927 21:33:46.226373 10591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f8760c6f12d4916850047930a9c2e3d, State: Initialized, Role: LEARNER
I20260927 21:33:46.226683 10591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } }
I20260927 21:33:46.227490 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.228186 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:46.228289 10591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:46.243896 10605 catalog_manager.cc:1259] Loaded cluster ID: 795d1935d3b44545a5685909570ec94f
I20260927 21:33:46.244001 10605 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: loading cluster ID for follower catalog manager: success
I20260927 21:33:46.247658 10605 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: acquiring CA information for follower catalog manager: success
I20260927 21:33:46.250463 10605 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 21:33:46.340888 10513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:46.341174 10513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:46.341325 10513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:46.348027 10513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:46.348160 10513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:46.348246 10513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:46.348335 10513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:46.348410 10513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:46.357832 10513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.252.keytab
--principal=oryx/127.3.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:46.358961 10513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:46.360126 10513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:46.373229 10610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:46.373406 10611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:46.373886 10513 server_base.cc:1034] running on GCE node
W20260927 21:33:46.373893 10613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:46.374672 10513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:46.375837 10513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:46.377063 10513 hybrid_clock.cc:648] HybridClock initialized: now 1790544826377009 us; error 46 us; skew 500 ppm
I20260927 21:33:46.377393 10513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:46.381393 10513 init.cc:377] Logged in from keytab as oryx/127.3.186.252@KRBTEST.COM (short username oryx)
I20260927 21:33:46.383548 10513 webserver.cc:460] Webserver started at http://127.3.186.252:38579/ using document root <none> and password file <none>
I20260927 21:33:46.384294 10513 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:46.384410 10513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:46.384684 10513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:46.388206 10513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/instance:
uuid: "210006c161624aa0800c5fb8bdf28f3b"
format_stamp: "Formatted at 2026-09-27 21:33:46 on dist-test-slave-pqr8"
server_key: "26b5c258712cb805498f99c4399c7964"
server_key_iv: "b3be67c0390bb75546e1dc8080e2ce54"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.389030 10513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal/instance:
uuid: "210006c161624aa0800c5fb8bdf28f3b"
format_stamp: "Formatted at 2026-09-27 21:33:46 on dist-test-slave-pqr8"
server_key: "26b5c258712cb805498f99c4399c7964"
server_key_iv: "b3be67c0390bb75546e1dc8080e2ce54"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.394838 10513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:33:46.398703 10620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:46.399873 10513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:33:46.399993 10513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
uuid: "210006c161624aa0800c5fb8bdf28f3b"
format_stamp: "Formatted at 2026-09-27 21:33:46 on dist-test-slave-pqr8"
server_key: "26b5c258712cb805498f99c4399c7964"
server_key_iv: "b3be67c0390bb75546e1dc8080e2ce54"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.400133 10513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:46.404425 10408 heartbeater.cc:503] Master 127.3.186.254:44169 was elected leader, sending a full tablet report...
I20260927 21:33:46.412256 10513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:46.415750 10513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:46.415997 10513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:46.436061 10513 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.252:36899
I20260927 21:33:46.436088 10671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.252:36899 every 8 connection(s)
I20260927 21:33:46.437309 10513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
I20260927 21:33:46.440680 10672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:46.441480 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10513
I20260927 21:33:46.443418 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.443703 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:46.451288 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.444081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55732)
0927 21:33:46.445179 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.445198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.445302 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.445936 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.445940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.445960 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:46.446104 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.446111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.447453 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.447459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.449305 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.449316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.450531 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.450537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.450726 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.450798 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.450893 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.451019 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973}
I20260927 21:33:46.451488 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.444851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.3.186.253:42241)
0927 21:33:46.445500 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.445540 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.445562 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:33:46.445648 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.445655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.445752 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.446209 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:46.446369 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.447323 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.449439 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.450375 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.450510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.450726 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.450979 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.451125 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":672,"threads_started":1}
I20260927 21:33:46.450832 10672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:33:46.452263 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ae9d85f2c54051a7c7a95f68009e8b. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.452497 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:46.453397 10510 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: attempting to promote NON_VOTER 8f8760c6f12d4916850047930a9c2e3d to VOTER
I20260927 21:33:46.454317 10510 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } }
I20260927 21:33:46.456157 10566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 LEARNER]: Refusing update from remote peer d1ae9d85f2c54051a7c7a95f68009e8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:33:46.456722 10511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.252@KRBTEST.COM
I20260927 21:33:46.459383 10510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } }
I20260927 21:33:46.460016 10566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } }
I20260927 21:33:46.463516 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } }
I20260927 21:33:46.463781 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:46.463980 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } }
I20260927 21:33:46.464221 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:46.465271 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } }
I20260927 21:33:46.465554 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.252@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:46.469008 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.443419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.3.186.252:36899)
0927 21:33:46.446026 (+ 2607us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.446044 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.446087 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:33:46.447835 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.447859 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.448332 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.448736 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.450402 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.452358 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.454251 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.454894 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.455071 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.460234 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.460248 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.460260 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.460304 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.464132 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.465109 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.465116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.465124 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.465291 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.465852 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.465861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.465867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.466523 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.466789 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.467436 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.467800 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2400,"thread_start_us":2224,"threads_started":1}
I20260927 21:33:46.472599 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.454112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.3.186.254:44169)
0927 21:33:46.454254 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.454259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.456447 (+ 2188us) server_negotiation.cc:407] Connection header received
0927 21:33:46.457230 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.457238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.457310 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.457403 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.458573 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.459717 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.462136 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.462907 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.462999 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.467172 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.467178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.467185 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.467224 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.470016 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.470833 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.470839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.470843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.470975 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.471398 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.471402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.471405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.471650 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.471787 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.472237 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.472375 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:33:46.473179 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.454030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:51556)
0927 21:33:46.456318 (+ 2288us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.456366 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.456562 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.457454 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.457461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.458076 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.458368 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.458382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.459909 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.459914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.461930 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.461946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.463255 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.463262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.463684 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.464337 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.464361 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.466997 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.470122 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.470131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.470143 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.470711 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.471096 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.471100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.471104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.471275 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.471813 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.471821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.471976 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.472152 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.472416 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":1909,"threads_started":1}
I20260927 21:33:46.474336 10672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.252@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
I20260927 21:33:46.484560 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.474911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55740)
0927 21:33:46.475080 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.475094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.475159 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.475388 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.475391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.475667 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.475772 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.475778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.476591 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.476594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.477434 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.477443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.478089 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.478093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.478212 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.478630 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.478645 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.480257 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.482925 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.482928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.482930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.483253 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.483541 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.483544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.483546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.483655 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.484027 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.484030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.484122 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.484217 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.484384 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:33:46.484694 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.474928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.3.186.253:42241)
0927 21:33:46.475109 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.475114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.475128 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:33:46.475239 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.475242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.475292 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.475362 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.475872 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.476490 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.477584 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.477943 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.478095 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.480397 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.480401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.480404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.480430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.482824 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.483340 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.483343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.483345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.483458 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.483735 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.483738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.483740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.483940 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.484040 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.484297 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.484450 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:33:46.485352 10672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.252@KRBTEST.COM for oryx/127.3.186.252@KRBTEST.COM
I20260927 21:33:46.496126 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.485846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.0.0.1:39406)
0927 21:33:46.486036 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.486056 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.486148 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.486372 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.486376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.486707 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.486816 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.486823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.487686 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.487694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.488433 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.488442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.489085 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.489090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.489280 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.489727 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.489741 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.491342 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.494097 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.494104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.494109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.494403 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.494705 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.494710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.494716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.494841 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.495247 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.495251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.495378 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.495517 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.495743 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:33:46.496183 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.486034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.3.186.252:36899)
0927 21:33:46.486153 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.486158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.486174 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:33:46.486219 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.486225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.486273 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.486361 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.486930 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.487549 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.488576 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.488945 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.489074 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.491488 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.491501 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.491505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.491535 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.493984 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.494509 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.494514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.494517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.494607 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.494931 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.494938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.494941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.495148 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.495277 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.495585 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.495864 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":41}
I20260927 21:33:46.500320 10672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Bootstrap starting.
I20260927 21:33:46.503238 10672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:46.504372 10672 log.cc:824] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:46.506719 10672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: No bootstrap required, opened a new log
I20260927 21:33:46.513628 10672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } }
I20260927 21:33:46.513857 10672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:46.513979 10672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210006c161624aa0800c5fb8bdf28f3b, State: Initialized, Role: FOLLOWER
I20260927 21:33:46.514454 10672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } }
I20260927 21:33:46.515151 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } } }
I20260927 21:33:46.515411 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:33:46.516029 10672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:46.523442 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10013
W20260927 21:33:46.525216 10692 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:33:46.525317 10692 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:33:46.528805 10513 master_runner.cc:428] Detected that this master 210006c161624aa0800c5fb8bdf28f3b is joining an existing cluster
I20260927 21:33:46.528910 10513 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.186.252:36899
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.193@KRBTEST.COM
I20260927 21:33:46.546533 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.524628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.3.186.193:46147)
0927 21:33:46.525916 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.525924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.525952 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:33:46.526020 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.526029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.526244 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.526381 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.527938 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.529467 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.531657 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.532388 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.532508 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.538355 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.538369 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.538381 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.538427 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.542391 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.543155 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.543164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.543167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.543303 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.543707 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.543712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.543715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.544074 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.544315 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.545875 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.546111 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":978,"threads_started":1}
I20260927 21:33:46.548111 10136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:46.557801 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.536472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.3.186.253:42241)
0927 21:33:46.536623 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.536628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.539917 (+ 3289us) server_negotiation.cc:407] Connection header received
0927 21:33:46.540324 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.540332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.540421 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.540493 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.542168 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.543473 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.546040 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.546916 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.547057 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.550338 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.550347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.550353 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.550396 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.554229 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.555098 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.555105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.555111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.555265 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.555753 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.555760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.555765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.556191 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.556354 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.557051 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.557314 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:46.559366 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.536602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55744)
0927 21:33:46.539843 (+ 3241us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.539857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.540136 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.540741 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.540748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.541581 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.541976 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.541991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.543691 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.543697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.545841 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.545861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.547647 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.547656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.547917 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.548708 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.548737 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.550129 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.554396 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.554403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.554409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.554956 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.555417 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.555424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.555429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.555624 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.556450 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.556458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.556709 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.558302 (+ 1593us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.558694 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3087,"thread_start_us":2896,"threads_started":1}
I20260927 21:33:46.560298 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.534220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.3.186.254:44169)
0927 21:33:46.534394 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.534400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.534573 (+ 173us) server_negotiation.cc:407] Connection header received
0927 21:33:46.534796 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.534802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.534872 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.534966 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.537007 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.538074 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.549261 (+ 11187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.550207 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.550317 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.553693 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.553701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.553709 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.553771 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.556882 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.557646 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.557653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.557658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.557827 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.558342 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.558350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.558355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.558744 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.558893 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.559590 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.559823 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:46.561928 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.533830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:51560)
0927 21:33:46.534472 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.534490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.534669 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.535006 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.535011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.535624 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.536833 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.536845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.538249 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.538255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.549057 (+ 10802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.549076 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.550640 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.550649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.550904 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.551842 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.551924 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.553411 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.557056 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.557062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.557067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.557484 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.557991 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.557997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.558002 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.558193 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.558981 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.558987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.559299 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.560532 (+ 1233us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.560986 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":577,"threads_started":1}
I20260927 21:33:46.576536 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.540037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.0.0.1:39408)
0927 21:33:46.559562 (+ 19525us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.559579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.559682 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.560276 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.560280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.560989 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.561680 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.561692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.563439 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.563448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.565004 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.565017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.569342 (+ 4325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.569350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.569612 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.570192 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.570218 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.570884 (+ 666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.574426 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.574433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.574436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.574755 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.575102 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.575106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.575113 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.575201 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.575628 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.575636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.575754 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.575865 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.576106 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19372,"thread_start_us":19159,"threads_started":1}
I20260927 21:33:46.576560 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.539911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.3.186.252:36899)
0927 21:33:46.540059 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.540064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.559656 (+ 19592us) server_negotiation.cc:407] Connection header received
0927 21:33:46.560002 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.560007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.560064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.560143 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.561804 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.562744 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.565910 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.566485 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.566549 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.571044 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.571049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.571054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.571478 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.574317 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.574862 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.574866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.574869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.574994 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.575302 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.575307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.575309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.575531 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.575679 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.575947 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.576217 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":41}
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.254@KRBTEST.COM for oryx/127.3.186.252@KRBTEST.COM
I20260927 21:33:46.596709 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.567263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.0.0.1:39420)
0927 21:33:46.567407 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.567422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.567514 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.572294 (+ 4780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.572300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.572707 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.572882 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.572892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.587117 (+ 14225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.587128 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.588019 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.588031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.589166 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.589172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.589335 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.589852 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.589869 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.591707 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.594561 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.594567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.594571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.594887 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.595236 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.595240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.595244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.595352 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.595812 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.595816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.595957 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.596098 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.596333 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:33:46.596757 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.567705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.3.186.252:36899)
0927 21:33:46.571603 (+ 3898us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.571607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.571626 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:33:46.572016 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.572024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.572105 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.573076 (+ 971us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.585891 (+ 12815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.586973 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.588181 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.589011 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.589153 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.591925 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.591930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.591933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.591959 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.594432 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.595006 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.595014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.595017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.595127 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.595449 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.595455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.595458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.595699 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.595832 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.596178 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.596346 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3773,"thread_start_us":2397,"threads_started":1}
I20260927 21:33:46.598172 9952 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.186.252:36899: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b"
member_type: NON_VOTER
last_known_addr {
host: "127.3.186.252"
port: 36899
}
attrs {
promote: true
}
}
dest_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b"
cas_config_opid_index: 6
I20260927 21:33:46.598773 9952 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } }
I20260927 21:33:46.601140 10566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Refusing update from remote peer d1ae9d85f2c54051a7c7a95f68009e8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:33:46.601472 10647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:46.604043 10647 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Refusing update from remote peer d1ae9d85f2c54051a7c7a95f68009e8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:33:46.604735 10510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 21:33:46.605225 10510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:46.610348 10510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.614194 10566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.619745 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.620101 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:33:46.622326 10510 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.186.252:36899
I20260927 21:33:46.622833 10513 master.cc:537] Master@127.3.186.252:36899 shutting down...
I20260927 21:33:46.622814 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.623174 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:46.623817 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.624042 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:46.629575 10647 raft_consensus.cc:858] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } }
W20260927 21:33:46.629912 10647 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } }
I20260927 21:33:46.641908 10647 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.645885 10709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.646252 10709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 21:33:46.646782 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ae9d85f2c54051a7c7a95f68009e8b. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.647008 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 21:33:46.647504 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: NON_VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: true } } }
I20260927 21:33:46.647708 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 21:33:46.648299 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.648586 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
W20260927 21:33:46.649739 9939 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 21:33:46.650080 10513 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 21:33:46.650548 10513 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 21:33:46.650759 10513 tablet_replica.cc:330] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: stopping tablet replica
I20260927 21:33:46.660200 10513 master.cc:552] Master@127.3.186.252:36899 shutdown complete.
I20260927 21:33:46.660288 10513 master_runner.cc:305] Clearing existing system tablet
I20260927 21:33:46.660746 10513 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:46.660826 10513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:46.665743 10513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:33:46.667191 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.666743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.0.0.1:39426)
0927 21:33:46.666890 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.667005 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:33:46.667636 9939 consensus_peers.cc:487] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b -> Peer 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252:36899): Couldn't send request to peer 210006c161624aa0800c5fb8bdf28f3b. Status: Network error: Client connection negotiation failed: client connection to 127.3.186.252:36899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:33:46.668227 10712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:46.668569 10513 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:46.668803 10513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 21:33:46.668903 10513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
uuid: "210006c161624aa0800c5fb8bdf28f3b"
format_stamp: "Formatted at 2026-09-27 21:33:46 on dist-test-slave-pqr8"
server_key: "26b5c258712cb805498f99c4399c7964"
server_key_iv: "b3be67c0390bb75546e1dc8080e2ce54"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.671804 10513 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:46.676023 10513 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:33:46.676169 10513 log.cc:1196] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 21:33:46.676571 10513 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Deleting consensus metadata
I20260927 21:33:46.676782 10513 master_runner.cc:315] Copying system tablet from 127.3.186.254:44169
I20260927 21:33:46.678913 10513 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: tablet copy: Beginning tablet copy session from remote peer at address 127.3.186.254:44169
I20260927 21:33:46.696138 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.679531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.3.186.254:44169)
0927 21:33:46.680107 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:46.680116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:46.686026 (+ 5910us) server_negotiation.cc:407] Connection header received
0927 21:33:46.686215 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:46.686220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:46.686298 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:46.686390 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:46.687151 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.687845 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.689015 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.689403 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.689546 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.691211 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:46.691217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:46.691222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:46.691257 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:46.693806 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.694379 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.694389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.694391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.694499 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:46.694842 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:46.694849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:46.694853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:46.695079 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:46.695218 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:46.695582 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 21:33:46.695789 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448}
I20260927 21:33:46.696161 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:46.679492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:51566)
0927 21:33:46.685929 (+ 6437us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:46.685943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:46.686083 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:46.686414 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:46.686418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:46.686845 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:46.687014 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.687022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.687980 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.687987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.688875 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:46.688885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:46.689567 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:46.689573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:46.689768 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:46.690362 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:46.690378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:46.691042 (+ 664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:46.693922 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.693929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.693933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.694267 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.694598 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:46.694603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:46.694607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:46.694742 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:46.695186 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:46.695191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:46.695337 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 21:33:46.695477 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 21:33:46.695752 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6299,"thread_start_us":5757,"threads_started":1}
I20260927 21:33:46.696949 9972 tablet_copy_service.cc:140] P d1ae9d85f2c54051a7c7a95f68009e8b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 210006c161624aa0800c5fb8bdf28f3b ({username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:51566)
I20260927 21:33:46.697105 9972 tablet_copy_service.cc:161] P d1ae9d85f2c54051a7c7a95f68009e8b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 210006c161624aa0800c5fb8bdf28f3b at {username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:51566: session id = 210006c161624aa0800c5fb8bdf28f3b-00000000000000000000000000000000
I20260927 21:33:46.699620 9972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:33:46.700924 10513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:33:46.705627 10513 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: tablet copy: Starting download of 0 data blocks...
I20260927 21:33:46.705899 10513 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: tablet copy: Starting download of 1 WAL segments...
I20260927 21:33:46.707002 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10013
I20260927 21:33:46.708544 10513 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:33:46.712476 9972 tablet_copy_service.cc:342] P d1ae9d85f2c54051a7c7a95f68009e8b: Request end of tablet copy session 210006c161624aa0800c5fb8bdf28f3b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:51566
I20260927 21:33:46.712591 9972 tablet_copy_service.cc:433] P d1ae9d85f2c54051a7c7a95f68009e8b: ending tablet copy session 210006c161624aa0800c5fb8bdf28f3b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 210006c161624aa0800c5fb8bdf28f3b
I20260927 21:33:46.723706 10513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:46.726739 10719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:46.726965 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:46.733383 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:46147
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:44169
--webserver_port=45415
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:46.734084 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:46.740334 10513 server_base.cc:1034] running on GCE node
I20260927 21:33:46.740655 10513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:46.741297 10513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:46.742504 10513 hybrid_clock.cc:648] HybridClock initialized: now 1790544826742461 us; error 30 us; skew 500 ppm
I20260927 21:33:46.742776 10513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:46 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544826, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:46.744870 10513 init.cc:377] Logged in from keytab as oryx/127.3.186.252@KRBTEST.COM (short username oryx)
I20260927 21:33:46.749819 10513 webserver.cc:460] Webserver started at http://127.3.186.252:43223/ using document root <none> and password file <none>
I20260927 21:33:46.750125 10513 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:46.750237 10513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:46.753139 10513 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:46.754910 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:46.755288 10513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:33:46.755457 10513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
uuid: "210006c161624aa0800c5fb8bdf28f3b"
format_stamp: "Formatted at 2026-09-27 21:33:46 on dist-test-slave-pqr8"
server_key: "26b5c258712cb805498f99c4399c7964"
server_key_iv: "b3be67c0390bb75546e1dc8080e2ce54"
server_key_version: "encryptionkey@0"
I20260927 21:33:46.755705 10513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:46.765829 10513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:46.769567 10513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:46.790374 10513 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.252:36899
I20260927 21:33:46.790439 10781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.252:36899 every 8 connection(s)
I20260927 21:33:46.791460 10513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
I20260927 21:33:46.796244 10782 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:33:46.799803 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Bootstrap starting.
I20260927 21:33:46.814019 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:33:46.814474 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Bootstrap complete.
I20260927 21:33:46.815462 10782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } }
I20260927 21:33:46.815837 10782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 210006c161624aa0800c5fb8bdf28f3b, State: Initialized, Role: LEARNER
I20260927 21:33:46.816110 10782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } }
I20260927 21:33:46.817107 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:46.817416 10782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:46.817459 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:46.834174 10796 catalog_manager.cc:1259] Loaded cluster ID: 795d1935d3b44545a5685909570ec94f
I20260927 21:33:46.834271 10796 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: loading cluster ID for follower catalog manager: success
I20260927 21:33:46.836521 10796 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: acquiring CA information for follower catalog manager: success
I20260927 21:33:46.838363 10796 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:33:47.065120 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:47.065433 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:47.072564 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.065758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.3.186.252:36899)
0927 21:33:47.066354 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.066361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.066381 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:33:47.066460 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.066464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.066543 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.066691 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:47.067071 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.068128 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.070937 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.071684 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.071781 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.071936 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.072235 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.072323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"spinlock_wait_cycles":1408,"thread_start_us":423,"threads_started":1}
I20260927 21:33:47.072597 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.065482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.0.0.1:39428)
0927 21:33:47.066147 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.066169 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.066299 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.066708 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.066712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.066739 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:47.066929 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.066946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.068341 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.068347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.070795 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.070810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.071841 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.071847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.072073 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.072109 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.072206 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.072335 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504}
I20260927 21:33:47.073438 10757 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 21:33:47.073889 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1ae9d85f2c54051a7c7a95f68009e8b. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: NON_VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: true } } }
I20260927 21:33:47.074147 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: LEARNER
I20260927 21:33:47.074416 10679 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: attempting to promote NON_VOTER 210006c161624aa0800c5fb8bdf28f3b to VOTER
I20260927 21:33:47.074999 10679 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } }
I20260927 21:33:47.076694 10757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 LEARNER]: Refusing update from remote peer d1ae9d85f2c54051a7c7a95f68009e8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 21:33:47.076858 10566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Refusing update from remote peer d1ae9d85f2c54051a7c7a95f68009e8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 21:33:47.077279 10511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:33:47.077621 10679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:33:47.079996 10679 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.084969 10566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.089810 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.090152 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:33:47.090909 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.091078 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:47.091464 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.091675 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:47.092298 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.092506 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:47.092794 10757 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.095135 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:47.095372 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:33:47.098757 10724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:47.099012 10724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:47.099169 10724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:47.106227 10724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:47.106333 10724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:47.106509 10724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:47.115612 10724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:46147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=45415
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:47.116714 10724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:47.117961 10724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:47.130447 10805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:47.130616 10804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:47.131165 10807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:47.131494 10724 server_base.cc:1034] running on GCE node
I20260927 21:33:47.132437 10724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:47.133246 10724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:47.134471 10724 hybrid_clock.cc:648] HybridClock initialized: now 1790544827134405 us; error 62 us; skew 500 ppm
I20260927 21:33:47.134776 10724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:47.147845 10724 init.cc:377] Logged in from keytab as oryx/127.3.186.193@KRBTEST.COM (short username oryx)
I20260927 21:33:47.149145 10724 webserver.cc:460] Webserver started at http://127.3.186.193:45415/ using document root <none> and password file <none>
I20260927 21:33:47.149804 10724 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:47.149907 10724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:47.154594 10724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:33:47.157034 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:47.158267 10724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:33:47.158384 10724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "3c8b773ec5984807a386ae0394a7472d"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "b7c5d5e81b4a9b28889f09108347776b"
server_key_iv: "203eccabe9d80fe20b7a238f5f6f40ea"
server_key_version: "encryptionkey@0"
I20260927 21:33:47.159216 10724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:47.190608 10724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:47.193809 10724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:47.194005 10724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:47.194814 10724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:47.196031 10724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:47.196106 10724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:47.196267 10724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:47.196353 10724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:47.230566 10724 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:46147
I20260927 21:33:47.230659 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:46147 every 8 connection(s)
I20260927 21:33:47.231848 10724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:47.237087 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10724
I20260927 21:33:47.237486 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10147
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.194@KRBTEST.COM
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.252@KRBTEST.COM
I20260927 21:33:47.269588 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.239075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.3.186.194:34727)
0927 21:33:47.246007 (+ 6932us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.246015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.246052 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:33:47.246132 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.246140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.246351 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.248926 (+ 2575us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.249148 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.251099 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.252904 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.253718 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.253841 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.260290 (+ 6449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.260300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.260313 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.260351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.265731 (+ 5380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.266614 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.266624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.266628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.266768 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267196 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.267597 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.267862 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.268931 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.269187 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6759,"thread_start_us":6616,"threads_started":1}
I20260927 21:33:47.269860 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.250778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:46053 (local address 127.3.186.252:36899)
0927 21:33:47.250979 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.250984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.252497 (+ 1513us) server_negotiation.cc:407] Connection header received
0927 21:33:47.252684 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.252690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.252762 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.252840 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.253862 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.254887 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.257144 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.257766 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.257859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.262524 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.262531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.262535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.262567 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.265873 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267127 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.267285 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267903 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.268213 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.268322 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.269272 (+ 950us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.269482 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 21:33:47.269882 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.234552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:42171 (local address 127.3.186.254:44169)
0927 21:33:47.241838 (+ 7286us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.241842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.250562 (+ 8720us) server_negotiation.cc:407] Connection header received
0927 21:33:47.250785 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.250790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.250911 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.250983 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.252631 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.253714 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.256380 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.257116 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.257216 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.262501 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.262508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.262512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.262558 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.266128 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267127 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.267402 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267903 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.268220 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.268322 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.269275 (+ 953us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.269476 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7192,"thread_start_us":7108,"threads_started":1}
I20260927 21:33:47.270526 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.234434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:53403 (local address 127.3.186.253:42241)
0927 21:33:47.249720 (+ 15286us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.249726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.249744 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:33:47.249793 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.249798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.249873 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.249986 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.252682 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.253714 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.256752 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.257378 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.257461 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.262248 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.262256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.262265 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.262304 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.266356 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267308 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.267454 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267926 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.267930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.267933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.268227 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.268328 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.269265 (+ 937us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.269448 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15154,"thread_start_us":15007,"threads_started":1}
I20260927 21:33:47.270709 10270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:47.272409 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.251098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.3.186.193:46053)
0927 21:33:47.252388 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.252407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.252521 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.253018 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.253022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.253444 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:47.253727 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.253736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.255036 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.255041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.257000 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.257012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.258358 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.258362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.258466 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.260215 (+ 1749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:47.260233 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:47.262340 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:47.266038 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.266047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.266061 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.266989 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.267571 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.267579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.267774 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.268428 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:47.268435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:47.268848 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.270490 (+ 1642us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.270767 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":37,"spinlock_wait_cycles":41216,"thread_start_us":1071,"threads_started":1}
I20260927 21:33:47.272670 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.235038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.3.186.193:53403)
0927 21:33:47.246125 (+ 11087us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.246166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.249052 (+ 2886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.250279 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.250294 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.251110 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:47.252123 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.252153 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.253877 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.253881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.256223 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.256236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.257783 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.257790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.257976 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.258698 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:47.258737 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:47.262000 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:47.266539 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.266543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.266547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.267189 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.267611 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.267622 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.267802 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.268429 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:47.268435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:47.268868 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.270750 (+ 1882us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.271013 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10810,"mutex_wait_us":549,"spinlock_wait_cycles":39296,"thread_start_us":15066,"threads_started":1}
I20260927 21:33:47.274092 10927 heartbeater.cc:347] Connected to a master server at 127.3.186.252:36899
I20260927 21:33:47.274161 10929 heartbeater.cc:347] Connected to a master server at 127.3.186.253:42241
I20260927 21:33:47.274422 10929 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:47.274428 10927 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:47.275420 10929 heartbeater.cc:511] Master 127.3.186.253:42241 requested a full tablet report, sending...
I20260927 21:33:47.275449 10927 heartbeater.cc:511] Master 127.3.186.252:36899 requested a full tablet report, sending...
I20260927 21:33:47.276930 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.235038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:42171)
0927 21:33:47.250467 (+ 15429us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.250483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.250648 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.251199 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.251205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.251790 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:47.252509 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.252522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.253877 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.253881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.256215 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.256230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.257588 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.257595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.257882 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.258818 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:47.258845 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:47.262301 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:47.266321 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.266325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.266329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.266989 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.267508 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.267515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.267522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.267774 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.268428 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:47.268435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:47.268848 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.270098 (+ 1250us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.276503 (+ 6405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15278,"mutex_wait_us":380,"spinlock_wait_cycles":75392,"thread_start_us":10729,"threads_started":1}
I20260927 21:33:47.278249 10556 ts_manager.cc:194] Registered new tserver with Master: 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147)
I20260927 21:33:47.279095 10928 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44169
I20260927 21:33:47.279230 10928 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:47.279556 10928 heartbeater.cc:511] Master 127.3.186.254:44169 requested a full tablet report, sending...
I20260927 21:33:47.279217 10747 ts_manager.cc:194] Registered new tserver with Master: 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147)
I20260927 21:33:47.280499 9952 ts_manager.cc:194] Re-registered known tserver with Master: 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147)
I20260927 21:33:47.281298 9952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:42171
I20260927 21:33:47.359330 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10147
I20260927 21:33:47.374945 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:34727
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:44169
--webserver_port=35711
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:47.680264 10939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:47.680529 10939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:47.680696 10939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:47.687492 10939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:47.687626 10939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:47.687795 10939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:47.696403 10939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:34727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=35711
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:47.697494 10939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:47.698668 10939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:47.710985 10947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:47.711093 10944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:47.711064 10945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:47.712018 10939 server_base.cc:1034] running on GCE node
I20260927 21:33:47.712579 10939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:47.713330 10939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:47.714541 10939 hybrid_clock.cc:648] HybridClock initialized: now 1790544827714483 us; error 67 us; skew 500 ppm
I20260927 21:33:47.714856 10939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:47.717782 10939 init.cc:377] Logged in from keytab as oryx/127.3.186.194@KRBTEST.COM (short username oryx)
I20260927 21:33:47.719149 10939 webserver.cc:460] Webserver started at http://127.3.186.194:35711/ using document root <none> and password file <none>
I20260927 21:33:47.719746 10939 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:47.719905 10939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:47.724750 10939 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:33:47.727536 10954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:47.728783 10939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:33:47.728909 10939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "c9c8e6fc0592439789a1a76296c757c2"
format_stamp: "Formatted at 2026-09-27 21:33:44 on dist-test-slave-pqr8"
server_key: "aa24ad900bbab055f9f731587ae68f99"
server_key_iv: "e6a9cae2cdbcbb8372461747a9f2c738"
server_key_version: "encryptionkey@0"
I20260927 21:33:47.729974 10939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:47.743778 10939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:47.747052 10939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:47.747255 10939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:47.748210 10939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:47.750028 10939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:47.750145 10939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:47.750247 10939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:47.750360 10939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:47.786172 10939 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:34727
I20260927 21:33:47.786253 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:34727 every 8 connection(s)
I20260927 21:33:47.787879 10939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:47.793632 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 10939
I20260927 21:33:47.793860 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10280
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.195@KRBTEST.COM
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.252@KRBTEST.COM
I20260927 21:33:47.846048 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.794773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.3.186.195:37289)
0927 21:33:47.820430 (+ 25657us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.820441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.821279 (+ 838us) server_negotiation.cc:407] Connection header received
0927 21:33:47.821368 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.821379 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.821664 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.821811 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.824595 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.827461 (+ 2866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.830336 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.831210 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.831316 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.837652 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.837667 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.837682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.837728 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.842121 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.842909 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.842918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.842921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.843075 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.843517 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.843522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.843525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.843942 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.844253 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.845370 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.845655 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25226,"thread_start_us":25331,"threads_started":1}
I20260927 21:33:47.847281 10403 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:47.860255 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.792897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.3.186.194:60971)
0927 21:33:47.806187 (+ 13290us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.806240 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.809207 (+ 2967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.827903 (+ 18696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.827914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.828747 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:47.829535 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.829560 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.831760 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.831766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.833997 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.834013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.836566 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.836574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.836873 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.837573 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:47.837603 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:47.842245 (+ 4642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:47.853141 (+ 10896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.853154 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.853172 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.853883 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.855552 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.855559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.855565 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.855838 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.856812 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:47.856823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:47.857317 (+ 494us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.857688 (+ 371us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.858336 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12888,"thread_start_us":12661,"threads_started":1}
Sep 27 21:33:47 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544827, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
I20260927 21:33:47.863924 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.793562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:60971 (local address 127.3.186.252:36899)
0927 21:33:47.826674 (+ 33112us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.826682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.826704 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:33:47.826759 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.826764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.826848 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.828273 (+ 1425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.830539 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.831572 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.834203 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.834888 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.837266 (+ 2378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.842468 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.842474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.842479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.842511 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.849337 (+ 6826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.855215 (+ 5878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.855223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.855228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.855408 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.856006 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.856013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.856019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.856451 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.863032 (+ 6581us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.863171 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.863410 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32912,"thread_start_us":32739,"threads_started":1}
I20260927 21:33:47.864897 11067 heartbeater.cc:347] Connected to a master server at 127.3.186.252:36899
I20260927 21:33:47.865226 11067 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:47.866465 11067 heartbeater.cc:511] Master 127.3.186.252:36899 requested a full tablet report, sending...
I20260927 21:33:47.868023 10747 ts_manager.cc:194] Registered new tserver with Master: c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727)
I20260927 21:33:47.868270 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.811110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:35871)
0927 21:33:47.821891 (+ 10781us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.821907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.822001 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.827903 (+ 5902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.827914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.830108 (+ 2194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:47.830405 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.830418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.839724 (+ 9306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.839731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.841342 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.841356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.845687 (+ 4331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.845693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.845836 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.846409 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:47.846429 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:47.850374 (+ 3945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:47.865575 (+ 15201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.865579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.865583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.866117 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.866558 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.866562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.866566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.866695 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.867307 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:47.867313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:47.867462 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.867594 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.867850 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10655,"thread_start_us":10489,"threads_started":1}
I20260927 21:33:47.869226 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.811093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:35871 (local address 127.3.186.254:44169)
0927 21:33:47.826912 (+ 15819us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.826920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.826938 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:33:47.826991 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.826996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.827060 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.835030 (+ 7970us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.835193 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.836310 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.844342 (+ 8032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.844962 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.847778 (+ 2816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.850607 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.850616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.850624 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.850671 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.862866 (+ 12195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.866271 (+ 3405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.866278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.866283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.866413 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.866833 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.866837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.866840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.867169 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.867329 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.868646 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.868885 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15696,"thread_start_us":16039,"threads_started":1}
I20260927 21:33:47.870002 11068 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44169
I20260927 21:33:47.870103 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.826986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.3.186.194:53227)
0927 21:33:47.839467 (+ 12481us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:47.839484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:47.839580 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:47.851973 (+ 12393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:47.851981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:47.852736 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:47.853162 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.853177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.854977 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.854985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.857245 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:47.857269 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.858812 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.858821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:47.859087 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.859740 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:47.859769 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:47.863404 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:47.867257 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.867262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.867265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.867681 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.868050 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:47.868055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:47.868058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:47.868296 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:47.869126 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:47.869131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:47.869263 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 21:33:47.869393 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 21:33:47.869678 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12347,"thread_start_us":12215,"threads_started":1}
I20260927 21:33:47.870555 11068 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:47.870857 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:47.827423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:53227 (local address 127.3.186.253:42241)
0927 21:33:47.851624 (+ 24201us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:47.851634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:47.851656 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:33:47.851724 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:47.851730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:47.851818 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:47.851925 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:47.853390 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.854774 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.857483 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:47.858358 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:47.858500 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:47.863663 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:47.863673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:47.863680 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:47.863730 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:47.867127 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.867801 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.867806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.867809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.867928 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:47.868620 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:47.868625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:47.868629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:47.868974 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:47.869108 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:47.869697 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 21:33:47.869908 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24060,"thread_start_us":23903,"threads_started":1}
I20260927 21:33:47.870988 11068 heartbeater.cc:511] Master 127.3.186.254:44169 requested a full tablet report, sending...
I20260927 21:33:47.871855 9952 ts_manager.cc:194] Re-registered known tserver with Master: c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727)
I20260927 21:33:47.872562 9952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:35871
I20260927 21:33:47.873054 11073 heartbeater.cc:347] Connected to a master server at 127.3.186.253:42241
I20260927 21:33:47.873167 11073 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:47.873637 11073 heartbeater.cc:511] Master 127.3.186.253:42241 requested a full tablet report, sending...
I20260927 21:33:47.874603 10556 ts_manager.cc:194] Registered new tserver with Master: c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727)
I20260927 21:33:47.943728 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10280
I20260927 21:33:47.959885 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:37289
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:44169
--webserver_port=42237
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:48.262677 11080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:48.262902 11080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:48.262979 11080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:48.269505 11080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:48.269603 11080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:48.269719 11080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:48.278358 11080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:37289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=42237
--tserver_master_addrs=127.3.186.254:44169,127.3.186.253:42241,127.3.186.252:36899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:48.279769 11080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:48.281255 11080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:33:48.283345 10928 heartbeater.cc:503] Master 127.3.186.254:44169 was elected leader, sending a full tablet report...
W20260927 21:33:48.294310 11085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:48.294356 11086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:48.294376 11088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:48.295029 11080 server_base.cc:1034] running on GCE node
I20260927 21:33:48.295552 11080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:48.296382 11080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:48.297590 11080 hybrid_clock.cc:648] HybridClock initialized: now 1790544828297549 us; error 36 us; skew 500 ppm
I20260927 21:33:48.297901 11080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:48 dist-test-slave-pqr8 krb5kdc[9907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544828, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:48.300812 11080 init.cc:377] Logged in from keytab as oryx/127.3.186.195@KRBTEST.COM (short username oryx)
I20260927 21:33:48.302062 11080 webserver.cc:460] Webserver started at http://127.3.186.195:42237/ using document root <none> and password file <none>
I20260927 21:33:48.302649 11080 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:48.302789 11080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:48.307572 11080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 21:33:48.310101 11095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:48.311270 11080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:48.311422 11080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "7d833d302caa47a387a8ae46f66f8824"
format_stamp: "Formatted at 2026-09-27 21:33:45 on dist-test-slave-pqr8"
server_key: "676d94166b390da2e1c1cf5c44a9ea18"
server_key_iv: "bedcf9002ed5ed52c397f15e5b1d7406"
server_key_version: "encryptionkey@0"
I20260927 21:33:48.312299 11080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:48.354225 11080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:48.358136 11080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:48.358382 11080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:48.359251 11080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:33:48.360466 11080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:48.360543 11080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:48.360684 11080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:48.360773 11080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:48.392248 11080 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:37289
I20260927 21:33:48.392318 11207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:37289 every 8 connection(s)
I20260927 21:33:48.393429 11080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:48.396801 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 11080
Sep 27 21:33:48 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544828, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.252@KRBTEST.COM
Sep 27 21:33:48 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544828, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:48.450474 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.396127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:59191 (local address 127.3.186.254:44169)
0927 21:33:48.431441 (+ 35314us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.431447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.431465 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:33:48.431518 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.431525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.431607 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.431727 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.433455 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.435517 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.437384 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.438142 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.438260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.442398 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.442404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.442408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.442441 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.446479 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.447328 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.447333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.447337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.447480 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.447951 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.447955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.447958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.448298 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.448421 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.449821 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.450043 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35203,"thread_start_us":36546,"threads_started":1}
I20260927 21:33:48.451396 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.397289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:59191)
0927 21:33:48.422151 (+ 24862us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.422180 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.428585 (+ 6405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.431890 (+ 3305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.431898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.432308 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:48.432588 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.432599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.435698 (+ 3099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.435706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.437224 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.437236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.438507 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.438514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.438709 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.439316 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:48.439337 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:48.442196 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:48.446657 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:48.446663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:48.446667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:48.447228 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:48.447587 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:48.447593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:48.447598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:48.447857 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:48.449005 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:48.449014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:48.449395 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.449727 (+ 332us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.450155 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24746,"mutex_wait_us":92,"thread_start_us":23922,"threads_started":1}
I20260927 21:33:48.453284 11209 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44169
I20260927 21:33:48.453647 11209 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:48.453871 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.396158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:41199 (local address 127.3.186.252:36899)
0927 21:33:48.411719 (+ 15561us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.411729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.421755 (+ 10026us) server_negotiation.cc:407] Connection header received
0927 21:33:48.429048 (+ 7293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.429056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.429152 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.429252 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.431936 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.433026 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.435793 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.436458 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.436552 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.442663 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.442673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.442680 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.442729 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.446315 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.451791 (+ 5476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.451796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.451799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.451932 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.452337 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.452343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.452346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.452686 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.452797 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.453278 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.453469 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15399,"thread_start_us":15560,"threads_started":1}
I20260927 21:33:48.454656 11209 heartbeater.cc:511] Master 127.3.186.254:44169 requested a full tablet report, sending...
I20260927 21:33:48.454741 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.396811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.3.186.195:41199)
0927 21:33:48.421601 (+ 24790us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.421644 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.428886 (+ 7242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.429797 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.429817 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.430903 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:48.431751 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.431775 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.433321 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.433371 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.435637 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.435653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.436904 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.436912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.437210 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.437995 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:48.438029 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:48.442439 (+ 4410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:48.446481 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:48.446490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:48.446505 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:48.447491 (+ 986us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:48.452081 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:48.452085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:48.452089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:48.452223 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:48.452861 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:48.452865 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:48.452992 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.454040 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.454292 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24498,"mutex_wait_us":2,"thread_start_us":25127,"threads_started":1}
I20260927 21:33:48.456370 11208 heartbeater.cc:347] Connected to a master server at 127.3.186.252:36899
I20260927 21:33:48.456377 9952 ts_manager.cc:194] Re-registered known tserver with Master: 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289)
I20260927 21:33:48.456503 11208 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:48.456884 11208 heartbeater.cc:511] Master 127.3.186.252:36899 requested a full tablet report, sending...
I20260927 21:33:48.457438 9952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:59191
I20260927 21:33:48.457909 10747 ts_manager.cc:194] Registered new tserver with Master: 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289)
I20260927 21:33:48.468583 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.421205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.3.186.253:42241)
0927 21:33:48.445011 (+ 23806us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.445017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.445034 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:33:48.445090 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.445097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.445167 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.445269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.449222 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.450361 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.452695 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.453341 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.453481 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.461034 (+ 7553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.461040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.461044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.461110 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.465545 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.466381 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.466387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.466394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.466575 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.466977 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.466982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.466985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.467289 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.467502 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.467976 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.468196 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23681,"thread_start_us":25741,"threads_started":1}
I20260927 21:33:48.470563 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.432850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.3.186.254:44169)
0927 21:33:48.436343 (+ 3493us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.436351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.436372 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:33:48.436432 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.436440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.436527 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.436661 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.444728 (+ 8067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.445850 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.448005 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.448688 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.448825 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.463265 (+ 14440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.463272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.463276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.463308 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.466616 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.468137 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.468143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.468147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.468265 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.468681 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.468688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.468693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.468992 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.469236 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.470019 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.470211 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3313,"thread_start_us":4271,"threads_started":1}
Sep 27 21:33:48 dist-test-slave-pqr8 krb5kdc[9907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544828, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.253@KRBTEST.COM
I20260927 21:33:48.476790 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.435853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.3.186.252:36899)
0927 21:33:48.440174 (+ 4321us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.440180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.447923 (+ 7743us) server_negotiation.cc:407] Connection header received
0927 21:33:48.447970 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.447974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.448049 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.448140 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.456766 (+ 8626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.457679 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.465797 (+ 8118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.466448 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.466539 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.471088 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.471095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.471101 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.471133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.474209 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.475029 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.475034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.475038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.475162 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.475501 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.475505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.475508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.475737 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.475952 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.476245 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.476446 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4196,"thread_start_us":3341,"threads_started":1}
I20260927 21:33:48.478694 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.447112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:55971 (local address 127.3.186.253:42241)
0927 21:33:48.463667 (+ 16555us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.463671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.463685 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:33:48.463718 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.463721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.463765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.463875 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.464654 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.465853 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.467753 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.468483 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.469186 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.472744 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.472748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.472751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.472776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.476110 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.476859 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.476866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.476869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.476994 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.477411 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.477415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.477417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.477703 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.477834 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.478161 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.478326 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16440,"thread_start_us":16320,"threads_started":1}
I20260927 21:33:48.478693 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.422476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.3.186.195:55971)
0927 21:33:48.429349 (+ 6873us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.429370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.429459 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.463899 (+ 34440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.463904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.464336 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:48.464503 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.464510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.465987 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.465992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.467543 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.467559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.468707 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.468715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.468902 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.469746 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:48.469772 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:48.472325 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:48.476254 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:48.476260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:48.476263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:48.476705 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:48.477120 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:48.477125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:48.477129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:48.477280 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:48.477820 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:48.477825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:48.477953 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.478082 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.478324 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6771,"thread_start_us":7072,"threads_started":1}
I20260927 21:33:48.479666 11210 heartbeater.cc:347] Connected to a master server at 127.3.186.253:42241
I20260927 21:33:48.479804 11210 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:48.480145 11210 heartbeater.cc:511] Master 127.3.186.253:42241 requested a full tablet report, sending...
I20260927 21:33:48.481025 10556 ts_manager.cc:194] Registered new tserver with Master: 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289)
I20260927 21:33:48.475224 9952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:48.483382 9952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:48.514159 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.499691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:46147 (local address 127.0.0.1:45154)
0927 21:33:48.500677 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.500696 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.505649 (+ 4953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.506566 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.506573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.506637 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:48.507465 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.507479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.508984 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.508992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.511733 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.511745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.513431 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.513437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.513614 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.513645 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.513737 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.513870 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":740,"threads_started":1}
I20260927 21:33:48.514205 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.499886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.3.186.193:46147)
0927 21:33:48.500469 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.500478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.500843 (+ 365us) server_negotiation.cc:407] Connection header received
0927 21:33:48.505883 (+ 5040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.505894 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.506182 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.506359 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:48.507710 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.508842 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.511886 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.513236 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.513385 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.513615 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.513790 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.513941 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":264,"threads_started":1}
I20260927 21:33:48.516947 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.500954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.3.186.194:34727)
0927 21:33:48.508057 (+ 7103us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.508069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.508100 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:33:48.508208 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.508219 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.508475 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.508696 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:48.509845 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.511221 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.513625 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.515853 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.515998 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.516213 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.516507 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.516653 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6920,"thread_start_us":6812,"threads_started":1}
I20260927 21:33:48.516981 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.500747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:34727 (local address 127.0.0.1:49684)
0927 21:33:48.501759 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.501781 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.502326 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.508692 (+ 6366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.508697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.508720 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:48.509137 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.509147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.511341 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.511348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.513285 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.513294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.516080 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.516087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.516327 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.516359 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.516466 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.516576 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":26,"thread_start_us":672,"threads_started":1}
I20260927 21:33:48.518169 10862 tablet_service.cc:1467] Processing CreateTablet for tablet 1791b9e8c47348fca510e71c254a327f (DEFAULT_TABLE table=test-table [id=a145caaf3806411e8d62f2544eb41aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:48.519362 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1791b9e8c47348fca510e71c254a327f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:48.524066 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.501731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:37289 (local address 127.0.0.1:45868)
0927 21:33:48.511763 (+ 10032us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.511800 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.511906 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.512364 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.512369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.512398 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:48.512778 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.512787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.514565 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.514572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.516420 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.516434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.523331 (+ 6897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.523337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.523498 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.523527 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.523634 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.523741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9905,"mutex_wait_us":23,"thread_start_us":9747,"threads_started":1}
I20260927 21:33:48.524092 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.502022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.3.186.195:37289)
0927 21:33:48.502842 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.502850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.511928 (+ 9078us) server_negotiation.cc:407] Connection header received
0927 21:33:48.512013 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.512020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.512190 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.512355 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:48.512963 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.514434 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.521786 (+ 7352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.523121 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.523249 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.523404 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.523678 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.523812 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":486,"threads_started":1}
I20260927 21:33:48.524336 11002 tablet_service.cc:1467] Processing CreateTablet for tablet 1791b9e8c47348fca510e71c254a327f (DEFAULT_TABLE table=test-table [id=a145caaf3806411e8d62f2544eb41aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:48.525457 11002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1791b9e8c47348fca510e71c254a327f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:48.527616 11143 tablet_service.cc:1467] Processing CreateTablet for tablet 1791b9e8c47348fca510e71c254a327f (DEFAULT_TABLE table=test-table [id=a145caaf3806411e8d62f2544eb41aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:48.528813 11143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1791b9e8c47348fca510e71c254a327f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:48.534564 11245 tablet_bootstrap.cc:492] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Bootstrap starting.
I20260927 21:33:48.537456 11246 tablet_bootstrap.cc:492] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Bootstrap starting.
I20260927 21:33:48.538615 11245 tablet_bootstrap.cc:654] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:48.539652 11245 log.cc:824] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:48.540726 11246 tablet_bootstrap.cc:654] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:48.541783 11246 log.cc:824] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:48.542083 11245 tablet_bootstrap.cc:492] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: No bootstrap required, opened a new log
I20260927 21:33:48.542315 11245 ts_tablet_manager.cc:1397] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:33:48.543517 11249 tablet_bootstrap.cc:492] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Bootstrap starting.
I20260927 21:33:48.547047 11249 tablet_bootstrap.cc:654] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:48.547868 11246 tablet_bootstrap.cc:492] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: No bootstrap required, opened a new log
I20260927 21:33:48.548102 11246 ts_tablet_manager.cc:1397] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260927 21:33:48.548182 11249 log.cc:824] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:48.550885 11249 tablet_bootstrap.cc:492] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: No bootstrap required, opened a new log
I20260927 21:33:48.551120 11249 ts_tablet_manager.cc:1397] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:33:48.552233 11245 raft_consensus.cc:348] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.552619 11245 raft_consensus.cc:374] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:48.552711 11245 raft_consensus.cc:729] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8b773ec5984807a386ae0394a7472d, State: Initialized, Role: FOLLOWER
I20260927 21:33:48.553274 11245 consensus_queue.cc:260] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.558775 11245 ts_tablet_manager.cc:1428] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Time spent starting tablet: real 0.016s user 0.008s sys 0.004s
I20260927 21:33:48.559074 11246 raft_consensus.cc:348] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.559440 11246 raft_consensus.cc:374] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:48.559558 11246 raft_consensus.cc:729] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c8e6fc0592439789a1a76296c757c2, State: Initialized, Role: FOLLOWER
I20260927 21:33:48.560076 11246 consensus_queue.cc:260] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.561009 11068 heartbeater.cc:503] Master 127.3.186.254:44169 was elected leader, sending a full tablet report...
I20260927 21:33:48.560842 11249 raft_consensus.cc:348] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.561208 11249 raft_consensus.cc:374] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:48.561292 11249 raft_consensus.cc:729] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d833d302caa47a387a8ae46f66f8824, State: Initialized, Role: FOLLOWER
I20260927 21:33:48.562059 11246 ts_tablet_manager.cc:1428] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260927 21:33:48.561902 11249 consensus_queue.cc:260] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.567571 11209 heartbeater.cc:503] Master 127.3.186.254:44169 was elected leader, sending a full tablet report...
I20260927 21:33:48.568209 11249 ts_tablet_manager.cc:1428] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260927 21:33:48.616022 11076 tablet.cc:2369] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:48.662446 11252 raft_consensus.cc:482] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:48.662717 11252 raft_consensus.cc:504] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
W20260927 21:33:48.663309 11211 tablet.cc:2369] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:48.664309 11252 leader_election.cc:288] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147), 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289)
I20260927 21:33:48.671577 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.664701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:38293 (local address 127.3.186.193:46147)
0927 21:33:48.664927 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.664937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.665379 (+ 442us) server_negotiation.cc:407] Connection header received
0927 21:33:48.665687 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.665693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.665784 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.665856 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:48.666496 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.667875 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.669756 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.670737 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.670847 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.670894 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.671251 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.671328 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:33:48.671715 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.664641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:46147 (local address 127.3.186.194:38293)
0927 21:33:48.665263 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.665277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.665494 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.666004 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.666008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.666030 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:48.666365 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.666377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.667992 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.667998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.669618 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.669631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.670915 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.670933 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.671088 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.671131 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.671228 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.671346 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":331,"threads_started":1}
I20260927 21:33:48.672256 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.665600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:52709 (local address 127.3.186.195:37289)
0927 21:33:48.665795 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.665800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.666292 (+ 492us) server_negotiation.cc:407] Connection header received
0927 21:33:48.666419 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.666426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.666509 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.666583 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:48.667559 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.668475 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.670369 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.671319 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.671458 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.671526 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.671923 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.672057 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:33:48.672273 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.665343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:37289 (local address 127.3.186.194:52709)
0927 21:33:48.666185 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:48.666198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:48.666295 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:48.666665 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:48.666670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:48.666694 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:48.667411 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.667422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.668608 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.668612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.670216 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:48.670228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.671522 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.671527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:48.671709 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.671740 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:33:48.671877 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 21:33:48.672005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":583,"threads_started":1}
I20260927 21:33:48.672734 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1791b9e8c47348fca510e71c254a327f" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b773ec5984807a386ae0394a7472d" is_pre_election: true
I20260927 21:33:48.673204 10882 raft_consensus.cc:2454] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 0.
I20260927 21:33:48.673214 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1791b9e8c47348fca510e71c254a327f" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d833d302caa47a387a8ae46f66f8824" is_pre_election: true
I20260927 21:33:48.673769 11163 raft_consensus.cc:2454] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 0.
I20260927 21:33:48.673846 10956 leader_election.cc:302] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8b773ec5984807a386ae0394a7472d, c9c8e6fc0592439789a1a76296c757c2; no voters:
I20260927 21:33:48.674275 11252 raft_consensus.cc:2781] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:48.674417 11252 raft_consensus.cc:482] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:48.674527 11252 raft_consensus.cc:3037] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:48.676497 11252 raft_consensus.cc:504] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.676975 11252 leader_election.cc:288] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 election: Requested vote from peers 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147), 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289)
I20260927 21:33:48.677372 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1791b9e8c47348fca510e71c254a327f" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b773ec5984807a386ae0394a7472d"
I20260927 21:33:48.677484 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1791b9e8c47348fca510e71c254a327f" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d833d302caa47a387a8ae46f66f8824"
I20260927 21:33:48.677587 10882 raft_consensus.cc:3037] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:48.677696 11163 raft_consensus.cc:3037] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:48.679641 10882 raft_consensus.cc:2454] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 1.
I20260927 21:33:48.679975 10956 leader_election.cc:302] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8b773ec5984807a386ae0394a7472d, c9c8e6fc0592439789a1a76296c757c2; no voters:
I20260927 21:33:48.680092 11163 raft_consensus.cc:2454] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 1.
I20260927 21:33:48.680244 11252 raft_consensus.cc:2781] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:48.680639 11252 raft_consensus.cc:686] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 1 LEADER]: Becoming Leader. State: Replica: c9c8e6fc0592439789a1a76296c757c2, State: Running, Role: LEADER
I20260927 21:33:48.681105 11252 consensus_queue.cc:237] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } }
I20260927 21:33:48.685837 9952 catalog_manager.cc:5495] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "c9c8e6fc0592439789a1a76296c757c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } health_report { overall_health: UNKNOWN } } }
W20260927 21:33:48.740934 10930 tablet.cc:2369] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:48.745723 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.740390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.3.186.194:34727)
0927 21:33:48.740547 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.740551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.740731 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:33:48.740901 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.740908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.741016 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.741168 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:48.741495 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.742288 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.743938 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.744333 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.744444 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.744835 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:48.745259 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:48.745360 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.745424 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.745477 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:33:48.753489 11163 raft_consensus.cc:1261] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Refusing update from remote peer c9c8e6fc0592439789a1a76296c757c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:48.753484 10882 raft_consensus.cc:1261] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Refusing update from remote peer c9c8e6fc0592439789a1a76296c757c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:48.754357 11256 consensus_queue.cc:1035] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:48.754657 11252 consensus_queue.cc:1035] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:48.786703 9952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:48.786955 9952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:48.799299 9952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1ae9d85f2c54051a7c7a95f68009e8b: Sending DeleteTablet for 3 replicas of tablet 1791b9e8c47348fca510e71c254a327f
I20260927 21:33:48.800534 11143 tablet_service.cc:1514] Processing DeleteTablet for tablet 1791b9e8c47348fca510e71c254a327f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:48 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45868
I20260927 21:33:48.800611 11002 tablet_service.cc:1514] Processing DeleteTablet for tablet 1791b9e8c47348fca510e71c254a327f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:48 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:49684
I20260927 21:33:48.802419 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet 1791b9e8c47348fca510e71c254a327f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:48 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45154
I20260927 21:33:48.815611 11270 tablet_replica.cc:330] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: stopping tablet replica
I20260927 21:33:48.816100 11270 raft_consensus.cc:2229] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:48.816785 11270 raft_consensus.cc:2258] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:48.820369 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 9920
I20260927 21:33:48.821287 9982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:48.824790 11270 ts_tablet_manager.cc:1905] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:48.836440 11272 tablet_replica.cc:330] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: stopping tablet replica
I20260927 21:33:48.836834 11272 raft_consensus.cc:2229] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:48.837293 11272 raft_consensus.cc:2258] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:48.840844 11269 tablet_replica.cc:330] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: stopping tablet replica
I20260927 21:33:48.841305 11269 raft_consensus.cc:2229] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:48.841841 11272 ts_tablet_manager.cc:1905] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:48.841883 11269 raft_consensus.cc:2258] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:48.844241 11269 ts_tablet_manager.cc:1905] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:48.844949 11270 ts_tablet_manager.cc:1918] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:48.845119 11270 log.cc:1196] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/1791b9e8c47348fca510e71c254a327f
I20260927 21:33:48.845693 11270 ts_tablet_manager.cc:1939] T 1791b9e8c47348fca510e71c254a327f P c9c8e6fc0592439789a1a76296c757c2: Deleting consensus metadata
I20260927 21:33:48.851143 11269 ts_tablet_manager.cc:1918] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:48.851351 11269 log.cc:1196] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/1791b9e8c47348fca510e71c254a327f
I20260927 21:33:48.851779 11272 ts_tablet_manager.cc:1918] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:48.851915 11269 ts_tablet_manager.cc:1939] T 1791b9e8c47348fca510e71c254a327f P 7d833d302caa47a387a8ae46f66f8824: Deleting consensus metadata
I20260927 21:33:48.851920 11272 log.cc:1196] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/1791b9e8c47348fca510e71c254a327f
I20260927 21:33:48.852402 11272 ts_tablet_manager.cc:1939] T 1791b9e8c47348fca510e71c254a327f P 3c8b773ec5984807a386ae0394a7472d: Deleting consensus metadata
I20260927 21:33:48.859225 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.803469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.3.186.253:42241)
0927 21:33:48.803876 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.803885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.818902 (+ 15017us) server_negotiation.cc:407] Connection header received
0927 21:33:48.819108 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.819113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.819203 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.819292 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.824112 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.825130 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.834913 (+ 9783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.835535 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.835651 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.842505 (+ 6854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.842511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.842515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.842551 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.845886 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.857242 (+ 11356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.857247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.857250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.857403 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.857812 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.857818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.857821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.858089 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.858344 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.858688 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.858916 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260927 21:33:48.871718 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:48.819590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.3.186.252:36899)
0927 21:33:48.819839 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:48.819845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:48.859341 (+ 39496us) server_negotiation.cc:407] Connection header received
0927 21:33:48.859591 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:48.859598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:48.859681 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:48.859752 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:48.861246 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.862158 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.863532 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:48.863935 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:48.864060 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:48.866787 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:48.866792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:48.866795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:48.866823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:48.869255 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.869917 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.869922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.869925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.870056 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:48.870405 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:48.870409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:48.870414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:48.870649 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:48.870912 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:48.871188 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 21:33:48.871397 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:48.993196 9937 catalog_manager.cc:4841] TS 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147): tablet 1791b9e8c47348fca510e71c254a327f (table test-table [id=a145caaf3806411e8d62f2544eb41aea]) successfully deleted
I20260927 21:33:48.993469 9937 catalog_manager.cc:4841] TS 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289): tablet 1791b9e8c47348fca510e71c254a327f (table test-table [id=a145caaf3806411e8d62f2544eb41aea]) successfully deleted
I20260927 21:33:48.993661 9937 catalog_manager.cc:4841] TS c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727): tablet 1791b9e8c47348fca510e71c254a327f (table test-table [id=a145caaf3806411e8d62f2544eb41aea]) successfully deleted
I20260927 21:33:48.993875 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 9920
I20260927 21:33:49.034775 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:49.019432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.3.186.252:36899)
0927 21:33:49.019596 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:49.019601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:49.019870 (+ 269us) server_negotiation.cc:407] Connection header received
0927 21:33:49.019983 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:49.019989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:49.020069 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:49.020172 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:49.022106 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:49.023221 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:49.024961 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:49.025379 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:49.025507 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:49.028152 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:49.028158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:49.028163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:49.028203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:49.031595 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:49.032345 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:49.032351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:49.032354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:49.032566 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:49.033126 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:49.033133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:49.033137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:49.033574 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:49.033803 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:49.034270 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 21:33:49.034512 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:33:49.035000 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:49.018411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.3.186.253:42241)
0927 21:33:49.018589 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:49.018620 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:49.019486 (+ 866us) server_negotiation.cc:407] Connection header received
0927 21:33:49.019783 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:49.019791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:49.019878 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:49.019952 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:49.022310 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:49.022963 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:49.024853 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:49.025492 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:49.025634 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:49.028941 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:49.028948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:49.028954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:49.029006 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:49.031711 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:49.032641 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:49.032646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:49.032649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:49.032829 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:49.033355 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:49.033361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:49.033364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:49.033737 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:49.033928 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:49.034470 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 21:33:49.034678 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:33:49.779654 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:49.778857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:55827)
0927 21:33:49.779333 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:49.779466 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
W20260927 21:33:49.780035 11068 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44169 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:49.786988 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:49.785387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:49443)
0927 21:33:49.786522 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:49.786724 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":846,"threads_started":1}
W20260927 21:33:49.787595 11209 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44169 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:49.809262 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:49.808605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:34131)
0927 21:33:49.809032 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:49.809122 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
W20260927 21:33:49.809695 10928 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44169 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:50.299542 11283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1ae9d85f2c54051a7c7a95f68009e8b)
I20260927 21:33:50.299822 11283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } }
I20260927 21:33:50.301072 11283 leader_election.cc:288] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169), 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253:42241)
I20260927 21:33:50.302168 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52046)
0927 21:33:50.301873 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.301984 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":366,"threads_started":1}
W20260927 21:33:50.302469 10731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:50.304564 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.304143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52056)
0927 21:33:50.304303 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.304416 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:33:50.304746 10731 leader_election.cc:334] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169): Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:50.305385 11288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1ae9d85f2c54051a7c7a95f68009e8b)
I20260927 21:33:50.305613 11288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } }
I20260927 21:33:50.306792 11288 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169), 210006c161624aa0800c5fb8bdf28f3b (127.3.186.252:36899)
I20260927 21:33:50.308596 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55794)
0927 21:33:50.302378 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.302391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:50.302490 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:50.303021 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:50.303026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:50.303052 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:50.303314 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.303327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.304516 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.304521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.306004 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.306015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.307522 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.307531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.307809 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.307865 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:33:50.307974 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:33:50.308154 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":385,"thread_start_us":1630,"threads_started":1}
I20260927 21:33:50.310037 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.301432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.3.186.253:42241)
0927 21:33:50.302228 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.302236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.302780 (+ 544us) server_negotiation.cc:407] Connection header received
0927 21:33:50.302849 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.302852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.302902 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.303007 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:50.303485 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.304379 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.306136 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.307249 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.308849 (+ 1600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.308974 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.309053 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.309229 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":455,"threads_started":1}
I20260927 21:33:50.314519 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.306880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52058)
0927 21:33:50.313970 (+ 7090us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.314217 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6958,"threads_started":1}
I20260927 21:33:50.314420 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "210006c161624aa0800c5fb8bdf28f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8f8760c6f12d4916850047930a9c2e3d" is_pre_election: true
I20260927 21:33:50.314849 10566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210006c161624aa0800c5fb8bdf28f3b in term 1.
I20260927 21:33:50.315502 10734 leader_election.cc:302] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210006c161624aa0800c5fb8bdf28f3b, 8f8760c6f12d4916850047930a9c2e3d; no voters: d1ae9d85f2c54051a7c7a95f68009e8b
W20260927 21:33:50.315685 10540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:50.316039 11283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:33:50.316167 11283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1ae9d85f2c54051a7c7a95f68009e8b)
I20260927 21:33:50.316280 11283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:33:50.317426 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.316968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52064)
0927 21:33:50.317114 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.317302 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 21:33:50.317651 10540 leader_election.cc:334] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169): Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:50.319216 11283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } }
I20260927 21:33:50.319883 11283 leader_election.cc:288] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [CANDIDATE]: Term 2 election: Requested vote from peers d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169), 8f8760c6f12d4916850047930a9c2e3d (127.3.186.253:42241)
I20260927 21:33:50.320380 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.319935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52074)
0927 21:33:50.320111 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.320202 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 21:33:50.320335 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "210006c161624aa0800c5fb8bdf28f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8f8760c6f12d4916850047930a9c2e3d"
I20260927 21:33:50.320494 10566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:33:50.321300 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.320956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52080)
0927 21:33:50.321093 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.321160 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:33:50.321974 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.306967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:36899 (local address 127.0.0.1:39446)
0927 21:33:50.314870 (+ 7903us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.314891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:50.315019 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:50.316467 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:50.316471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:50.316503 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:50.316755 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.316778 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.318248 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.318252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.319570 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.319581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.321308 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.321317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.321563 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.321598 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:33:50.321684 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:33:50.321776 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7789,"thread_start_us":6699,"threads_started":1}
I20260927 21:33:50.322158 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.307368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.3.186.252:36899)
0927 21:33:50.316000 (+ 8632us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.316007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.316037 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:33:50.316115 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.316122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.316284 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.316444 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:50.316941 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.318130 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.319739 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.321131 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.321250 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.321370 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.321724 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.321850 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8500,"thread_start_us":8298,"threads_started":1}
W20260927 21:33:50.322206 10731 leader_election.cc:334] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169): Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
I20260927 21:33:50.322832 10757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f8760c6f12d4916850047930a9c2e3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "210006c161624aa0800c5fb8bdf28f3b" is_pre_election: true
I20260927 21:33:50.323150 10757 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f8760c6f12d4916850047930a9c2e3d in current term 2: Already voted for candidate 210006c161624aa0800c5fb8bdf28f3b in this term.
I20260927 21:33:50.323544 10566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210006c161624aa0800c5fb8bdf28f3b in term 2.
I20260927 21:33:50.323736 10543 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f8760c6f12d4916850047930a9c2e3d; no voters: 210006c161624aa0800c5fb8bdf28f3b, d1ae9d85f2c54051a7c7a95f68009e8b
I20260927 21:33:50.324054 11288 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:33:50.324287 10734 leader_election.cc:302] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 210006c161624aa0800c5fb8bdf28f3b, 8f8760c6f12d4916850047930a9c2e3d; no voters: d1ae9d85f2c54051a7c7a95f68009e8b
I20260927 21:33:50.324546 11283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:33:50.324931 11283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [term 2 LEADER]: Becoming Leader. State: Replica: 210006c161624aa0800c5fb8bdf28f3b, State: Running, Role: LEADER
I20260927 21:33:50.325270 11283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } }
I20260927 21:33:50.326766 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210006c161624aa0800c5fb8bdf28f3b. Latest consensus state: current_term: 2 leader_uuid: "210006c161624aa0800c5fb8bdf28f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:50.327165 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:50.327804 11295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:50.331384 11295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:50.332125 11295 catalog_manager.cc:1259] Loaded cluster ID: 795d1935d3b44545a5685909570ec94f
I20260927 21:33:50.332221 11295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:50.333426 11295 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:50.334230 11295 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Loaded TSK: 0
I20260927 21:33:50.335018 11295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:33:50.349764 10747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:50.356038 10747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:50.359813 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.359410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52108)
0927 21:33:50.359550 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.359652 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:33:50.359859 10566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [term 2 FOLLOWER]: Refusing update from remote peer 210006c161624aa0800c5fb8bdf28f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 21:33:50.360667 11293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 21:33:50.360854 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.360463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52118)
0927 21:33:50.360628 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.360720 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 21:33:50.361063 10731 consensus_peers.cc:487] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b -> Peer d1ae9d85f2c54051a7c7a95f68009e8b (127.3.186.254:44169): Couldn't send request to peer d1ae9d85f2c54051a7c7a95f68009e8b. Status: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:33:50.363178 11288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210006c161624aa0800c5fb8bdf28f3b. Latest consensus state: current_term: 2 leader_uuid: "210006c161624aa0800c5fb8bdf28f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:50.363433 11288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:33:50.364881 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "210006c161624aa0800c5fb8bdf28f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:50.365152 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:50.365789 11288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "210006c161624aa0800c5fb8bdf28f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1ae9d85f2c54051a7c7a95f68009e8b" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44169 } } peers { permanent_uuid: "8f8760c6f12d4916850047930a9c2e3d" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 42241 } attrs { promote: false } } peers { permanent_uuid: "210006c161624aa0800c5fb8bdf28f3b" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 36899 } attrs { promote: false } } }
I20260927 21:33:50.366045 11288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8760c6f12d4916850047930a9c2e3d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:33:50.379195 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.373148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:34727 (local address 127.0.0.1:49700)
0927 21:33:50.373263 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.373278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:50.373377 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:50.374011 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:50.374015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:50.374036 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:50.374316 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.374323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.375230 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.375237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.376807 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.376818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.378155 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.378165 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.378426 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.378480 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 21:33:50.378587 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:33:50.378781 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:33:50.379371 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.373232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.3.186.194:34727)
0927 21:33:50.373667 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.373673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.373690 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:33:50.373778 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.373782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.373861 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.373972 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:50.374442 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.375097 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.376969 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.377915 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.378048 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.378687 (+ 639us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.378755 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.378884 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":327,"threads_started":1}
I20260927 21:33:50.379976 11002 tablet_service.cc:1467] Processing CreateTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 (DEFAULT_TABLE table=test-table [id=98711c17259c42958d0387f490a1bb2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:50.380373 11002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e095e17a52fe4f44b008f92243e9ebd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:50.384266 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.371998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:37289 (local address 127.0.0.1:45872)
0927 21:33:50.372630 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.372647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:50.372753 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:50.373136 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:50.373140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:50.373165 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:50.373360 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.373370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.380645 (+ 7275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.380653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.382571 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.382584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.383526 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.383531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.383705 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.383736 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:33:50.383841 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:33:50.383986 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532}
I20260927 21:33:50.384302 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.372160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.3.186.195:37289)
0927 21:33:50.372630 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.372637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.372694 (+ 57us) server_negotiation.cc:407] Connection header received
0927 21:33:50.372897 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.372902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.372992 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.373483 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:50.373668 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.380522 (+ 6854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.382685 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.383384 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.383519 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.383617 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.383872 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.384015 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":281,"threads_started":1}
I20260927 21:33:50.385249 11143 tablet_service.cc:1467] Processing CreateTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 (DEFAULT_TABLE table=test-table [id=98711c17259c42958d0387f490a1bb2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:50.385684 11143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e095e17a52fe4f44b008f92243e9ebd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:50.386981 11305 tablet_bootstrap.cc:492] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Bootstrap starting.
I20260927 21:33:50.388141 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.373821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:46147 (local address 127.0.0.1:45170)
0927 21:33:50.377951 (+ 4130us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.377969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:50.378066 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:50.382150 (+ 4084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:50.382154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:50.382177 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:50.382483 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.382495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.383734 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.383738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.385464 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:50.385475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.387433 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.387440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:50.387601 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.387630 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:33:50.387729 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:33:50.387850 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4027,"thread_start_us":3867,"threads_started":1}
I20260927 21:33:50.388271 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.373930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.3.186.193:46147)
0927 21:33:50.381849 (+ 7919us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.381855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.381873 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:33:50.381935 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.381941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.382005 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.382107 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:50.382642 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.383626 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.385798 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.387236 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.387369 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.387467 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.387803 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.387958 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7811,"thread_start_us":7712,"threads_started":1}
I20260927 21:33:50.389045 11305 tablet_bootstrap.cc:654] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:50.389047 10862 tablet_service.cc:1467] Processing CreateTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 (DEFAULT_TABLE table=test-table [id=98711c17259c42958d0387f490a1bb2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:50.389446 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e095e17a52fe4f44b008f92243e9ebd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:50.390897 11305 tablet_bootstrap.cc:492] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: No bootstrap required, opened a new log
I20260927 21:33:50.391048 11305 ts_tablet_manager.cc:1397] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260927 21:33:50.391666 11305 raft_consensus.cc:348] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.391887 11305 raft_consensus.cc:374] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:50.391968 11305 raft_consensus.cc:729] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c8e6fc0592439789a1a76296c757c2, State: Initialized, Role: FOLLOWER
I20260927 21:33:50.392196 11305 consensus_queue.cc:260] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.399031 11305 ts_tablet_manager.cc:1428] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Time spent starting tablet: real 0.008s user 0.002s sys 0.007s
I20260927 21:33:50.404139 11309 tablet_bootstrap.cc:492] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Bootstrap starting.
I20260927 21:33:50.406273 11309 tablet_bootstrap.cc:654] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:50.407737 11307 tablet_bootstrap.cc:492] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Bootstrap starting.
I20260927 21:33:50.408458 11067 heartbeater.cc:503] Master 127.3.186.252:36899 was elected leader, sending a full tablet report...
I20260927 21:33:50.409341 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.400036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:37257)
0927 21:33:50.408953 (+ 8917us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.409091 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8797,"threads_started":1}
I20260927 21:33:50.409557 11307 tablet_bootstrap.cc:654] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:50.411535 11307 tablet_bootstrap.cc:492] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: No bootstrap required, opened a new log
I20260927 21:33:50.411667 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.411266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:53401)
0927 21:33:50.411447 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.411534 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:33:50.411672 11307 ts_tablet_manager.cc:1397] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 21:33:50.412274 11307 raft_consensus.cc:348] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.412441 11307 raft_consensus.cc:374] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:50.412554 11307 raft_consensus.cc:729] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d833d302caa47a387a8ae46f66f8824, State: Initialized, Role: FOLLOWER
I20260927 21:33:50.412793 11307 consensus_queue.cc:260] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.413447 11307 ts_tablet_manager.cc:1428] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:33:50.414252 11309 tablet_bootstrap.cc:492] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: No bootstrap required, opened a new log
I20260927 21:33:50.414386 11309 ts_tablet_manager.cc:1397] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 21:33:50.414598 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.413822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:33879)
0927 21:33:50.414268 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.414397 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"threads_started":1}
I20260927 21:33:50.415148 11309 raft_consensus.cc:348] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.415382 11309 raft_consensus.cc:374] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:50.415481 11309 raft_consensus.cc:729] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8b773ec5984807a386ae0394a7472d, State: Initialized, Role: FOLLOWER
I20260927 21:33:50.415858 11208 heartbeater.cc:503] Master 127.3.186.252:36899 was elected leader, sending a full tablet report...
I20260927 21:33:50.415766 11309 consensus_queue.cc:260] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.417162 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.416821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:35689)
0927 21:33:50.416953 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.417031 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:33:50.421633 11309 ts_tablet_manager.cc:1428] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 21:33:50.421917 10927 heartbeater.cc:503] Master 127.3.186.252:36899 was elected leader, sending a full tablet report...
I20260927 21:33:50.422441 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.416772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:36583)
0927 21:33:50.421993 (+ 5221us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.422161 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5075,"thread_start_us":4915,"threads_started":1}
I20260927 21:33:50.423511 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.423144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:35543)
0927 21:33:50.423313 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.423388 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 21:33:50.543287 11308 raft_consensus.cc:482] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:50.543478 11308 raft_consensus.cc:504] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.544034 11308 leader_election.cc:288] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289), 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147)
I20260927 21:33:50.544548 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095e17a52fe4f44b008f92243e9ebd9" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d833d302caa47a387a8ae46f66f8824" is_pre_election: true
I20260927 21:33:50.544584 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095e17a52fe4f44b008f92243e9ebd9" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b773ec5984807a386ae0394a7472d" is_pre_election: true
I20260927 21:33:50.544790 11163 raft_consensus.cc:2454] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 0.
I20260927 21:33:50.544831 10882 raft_consensus.cc:2454] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 0.
I20260927 21:33:50.545254 10956 leader_election.cc:302] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d833d302caa47a387a8ae46f66f8824, c9c8e6fc0592439789a1a76296c757c2; no voters:
I20260927 21:33:50.545547 11308 raft_consensus.cc:2781] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:50.545691 11308 raft_consensus.cc:482] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:50.545784 11308 raft_consensus.cc:3037] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:50.547737 11308 raft_consensus.cc:504] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.548198 11308 leader_election.cc:288] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 election: Requested vote from peers 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289), 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147)
I20260927 21:33:50.548621 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095e17a52fe4f44b008f92243e9ebd9" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d833d302caa47a387a8ae46f66f8824"
I20260927 21:33:50.548705 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e095e17a52fe4f44b008f92243e9ebd9" candidate_uuid: "c9c8e6fc0592439789a1a76296c757c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b773ec5984807a386ae0394a7472d"
I20260927 21:33:50.548810 11163 raft_consensus.cc:3037] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:50.548890 10882 raft_consensus.cc:3037] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:50.551685 11163 raft_consensus.cc:2454] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 1.
I20260927 21:33:50.551685 10882 raft_consensus.cc:2454] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c8e6fc0592439789a1a76296c757c2 in term 1.
I20260927 21:33:50.552030 10956 leader_election.cc:302] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8b773ec5984807a386ae0394a7472d, c9c8e6fc0592439789a1a76296c757c2; no voters:
I20260927 21:33:50.552263 11308 raft_consensus.cc:2781] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:50.552654 11308 raft_consensus.cc:686] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 1 LEADER]: Becoming Leader. State: Replica: c9c8e6fc0592439789a1a76296c757c2, State: Running, Role: LEADER
I20260927 21:33:50.552955 11308 consensus_queue.cc:237] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } }
I20260927 21:33:50.553661 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.553195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:55013)
0927 21:33:50.553365 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.553449 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:33:50.555872 10746 catalog_manager.cc:5495] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "c9c8e6fc0592439789a1a76296c757c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9c8e6fc0592439789a1a76296c757c2" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 34727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:50.582441 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.578163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.3.186.194:34727)
0927 21:33:50.578325 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.578329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.578344 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:33:50.578480 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.578484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.578593 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.578709 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:50.578997 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.579629 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.580882 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.581264 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.581368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.581831 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:50.581921 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:50.581988 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.582081 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.582133 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:33:50.584929 10882 raft_consensus.cc:1261] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Refusing update from remote peer c9c8e6fc0592439789a1a76296c757c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:50.584929 11163 raft_consensus.cc:1261] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Refusing update from remote peer c9c8e6fc0592439789a1a76296c757c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:50.585467 11317 consensus_queue.cc:1035] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8b773ec5984807a386ae0394a7472d" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:50.585772 11308 consensus_queue.cc:1035] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d833d302caa47a387a8ae46f66f8824" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:50.588070 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.587527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:45097)
0927 21:33:50.587771 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.587868 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260927 21:33:50.588824 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.588429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:40991)
0927 21:33:50.588639 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.588712 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 21:33:50.592434 11321 mvcc.cc:205] Tried to move back new op lower bound from 7334071626070765568 to 7334071625947316224. Current Snapshot: MvccSnapshot[applied={T|T < 7334071626070765568}]
I20260927 21:33:50.593782 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.593268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.193:34167)
0927 21:33:50.593495 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.593610 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 21:33:50.594180 11322 mvcc.cc:205] Tried to move back new op lower bound from 7334071626070765568 to 7334071625947316224. Current Snapshot: MvccSnapshot[applied={T|T < 7334071626070765568}]
I20260927 21:33:50.597391 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.597011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:42479)
0927 21:33:50.597169 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.597268 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:33:50.602486 10746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:50.602756 10746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:50.605235 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.604831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.195:43619)
0927 21:33:50.605019 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.605092 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 21:33:50.605436 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.604903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.3.186.194:53195)
0927 21:33:50.605207 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.605284 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260927 21:33:50.610407 10746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 210006c161624aa0800c5fb8bdf28f3b: Sending DeleteTablet for 3 replicas of tablet e095e17a52fe4f44b008f92243e9ebd9
I20260927 21:33:50.611583 11002 tablet_service.cc:1514] Processing DeleteTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:50 UTC) from {username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:49700
I20260927 21:33:50.611626 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:50 UTC) from {username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:49700
I20260927 21:33:50.611860 11143 tablet_service.cc:1514] Processing DeleteTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:50 UTC) from {username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:45872
I20260927 21:33:50.612175 11330 tablet_replica.cc:330] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: stopping tablet replica
I20260927 21:33:50.612221 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet e095e17a52fe4f44b008f92243e9ebd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:50 UTC) from {username='oryx', principal='oryx/127.3.186.252@KRBTEST.COM'} at 127.0.0.1:45170
I20260927 21:33:50.612692 11330 raft_consensus.cc:2229] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:50.613006 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10724
I20260927 21:33:50.613046 11330 raft_consensus.cc:2258] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:50.613941 11330 ts_tablet_manager.cc:1905] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:50.615602 11332 tablet_replica.cc:330] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: stopping tablet replica
I20260927 21:33:50.615753 11332 raft_consensus.cc:2229] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:50.615993 11332 raft_consensus.cc:2258] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:50.616993 11332 ts_tablet_manager.cc:1905] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:50.620343 11330 ts_tablet_manager.cc:1918] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:50.620471 11330 log.cc:1196] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/e095e17a52fe4f44b008f92243e9ebd9
I20260927 21:33:50.620898 11330 ts_tablet_manager.cc:1939] T e095e17a52fe4f44b008f92243e9ebd9 P c9c8e6fc0592439789a1a76296c757c2: Deleting consensus metadata
I20260927 21:33:50.621989 10732 catalog_manager.cc:4841] TS c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727): tablet e095e17a52fe4f44b008f92243e9ebd9 (table test-table [id=98711c17259c42958d0387f490a1bb2c]) successfully deleted
W20260927 21:33:50.622694 10732 catalog_manager.cc:4808] TS c9c8e6fc0592439789a1a76296c757c2 (127.3.186.194:34727): delete failed for tablet e095e17a52fe4f44b008f92243e9ebd9 because the tablet was not found. No further retry: Not found: Tablet not found: e095e17a52fe4f44b008f92243e9ebd9
I20260927 21:33:50.622905 11332 ts_tablet_manager.cc:1918] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:50.622995 11332 log.cc:1196] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/e095e17a52fe4f44b008f92243e9ebd9
I20260927 21:33:50.623371 11332 ts_tablet_manager.cc:1939] T e095e17a52fe4f44b008f92243e9ebd9 P 3c8b773ec5984807a386ae0394a7472d: Deleting consensus metadata
I20260927 21:33:50.624192 10732 catalog_manager.cc:4841] TS 3c8b773ec5984807a386ae0394a7472d (127.3.186.193:46147): tablet e095e17a52fe4f44b008f92243e9ebd9 (table test-table [id=98711c17259c42958d0387f490a1bb2c]) successfully deleted
I20260927 21:33:50.624563 11331 tablet_replica.cc:330] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: stopping tablet replica
I20260927 21:33:50.624732 11331 raft_consensus.cc:2229] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:50.624962 11331 raft_consensus.cc:2258] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:50.625744 11331 ts_tablet_manager.cc:1905] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:50.630002 11331 ts_tablet_manager.cc:1918] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:50.630137 11331 log.cc:1196] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/e095e17a52fe4f44b008f92243e9ebd9
I20260927 21:33:50.630510 11331 ts_tablet_manager.cc:1939] T e095e17a52fe4f44b008f92243e9ebd9 P 7d833d302caa47a387a8ae46f66f8824: Deleting consensus metadata
I20260927 21:33:50.631246 10732 catalog_manager.cc:4841] TS 7d833d302caa47a387a8ae46f66f8824 (127.3.186.195:37289): tablet e095e17a52fe4f44b008f92243e9ebd9 (table test-table [id=98711c17259c42958d0387f490a1bb2c]) successfully deleted
I20260927 21:33:50.636500 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.613804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.3.186.193:46147)
0927 21:33:50.613965 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.613971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.624275 (+ 10304us) server_negotiation.cc:407] Connection header received
0927 21:33:50.624453 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.624458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.624535 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.624632 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:50.626003 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.626665 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.627835 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.628316 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.628430 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.631153 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:50.631167 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:50.631190 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:50.631216 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:50.634047 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:50.634626 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:50.634632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:50.634635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:50.634742 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:50.635089 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:50.635094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:50.635097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:50.635406 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:50.635743 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.636004 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.636214 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:33:50.637032 10922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:50.725260 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10724
I20260927 21:33:50.742210 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10939
I20260927 21:33:50.754572 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.742968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.3.186.194:34727)
0927 21:33:50.743132 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.743139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.743191 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:33:50.743389 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.743394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.743465 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.743573 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:50.744769 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.745460 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.746662 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.747087 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.747223 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.749492 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:50.749635 (+ 143us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:50.749652 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:50.749681 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:50.752089 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:50.752681 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:50.752689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:50.752692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:50.752819 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:50.753149 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:50.753153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:50.753156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:50.753470 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:50.753767 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.754055 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.754284 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:33:50.755175 11062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:50.877609 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10939
I20260927 21:33:50.899709 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 11080
I20260927 21:33:50.912077 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.911327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52130)
0927 21:33:50.911740 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.911860 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260927 21:33:50.912238 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.900410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.3.186.195:37289)
0927 21:33:50.900870 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:50.900874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:50.900890 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:33:50.900955 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:50.900958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:50.901014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:50.901112 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:50.902389 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.903094 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.904251 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:50.904621 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:50.904705 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:50.907090 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:50.907104 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:50.907121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:50.907151 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:50.909846 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:50.910395 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:50.910401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:50.910404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:50.910511 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:50.910832 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:50.910837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:50.910841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:50.911230 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:50.911541 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:50.911816 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 21:33:50.911962 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260927 21:33:50.912840 11203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:50.913375 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:50.912987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44169 (local address 127.0.0.1:52134)
0927 21:33:50.913128 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:50.913224 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 21:33:51.003043 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 11080
I20260927 21:33:51.019546 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10413
I20260927 21:33:51.020589 10586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:51.130275 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10413
I20260927 21:33:51.150326 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 10513
I20260927 21:33:51.151262 10777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:51.264271 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:51.263841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:42241 (local address 127.0.0.1:55806)
0927 21:33:51.263984 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:51.264135 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:33:51.264262 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 10513
2026-09-27T21:33:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8998 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:51 dist-test-slave-pqr8 krb5kdc[11344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:51 dist-test-slave-pqr8 krb5kdc[11344](info): set up 2 sockets
Sep 27 21:33:51 dist-test-slave-pqr8 krb5kdc[11344](info): commencing operation
krb5kdc: starting...
W20260927 21:33:52.326382 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:52 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab.
Entry for principal oryx/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab.
I20260927 21:33:52.506713 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:33897
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab
--principal=oryx/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:52.804220 11357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:52.804493 11357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:52.804647 11357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:52.811347 11357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:52.811461 11357 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:33:52.811550 11357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:52.811666 11357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:52.811749 11357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:52.811831 11357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:52.820592 11357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:33897
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.254.keytab
--principal=oryx/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:52.821671 11357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:52.822863 11357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:52.833588 11362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:52.833619 11365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:52.833652 11363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:52.834395 11357 server_base.cc:1034] running on GCE node
I20260927 21:33:52.835095 11357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:52.836313 11357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:52.837617 11357 hybrid_clock.cc:648] HybridClock initialized: now 1790544832837498 us; error 57 us; skew 500 ppm
I20260927 21:33:52.837930 11357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:52 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544832, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:52.844234 11357 init.cc:377] Logged in from keytab as oryx/127.3.186.254@KRBTEST.COM (short username oryx)
I20260927 21:33:52.845630 11357 webserver.cc:460] Webserver started at http://127.3.186.254:35969/ using document root <none> and password file <none>
I20260927 21:33:52.846249 11357 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:52.846376 11357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:52.846676 11357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:52.849376 11357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "af83bb2f411041eb9343ae4a601c6688"
format_stamp: "Formatted at 2026-09-27 21:33:52 on dist-test-slave-pqr8"
server_key: "95aaa697bf33499727f19f2e8e3d2646"
server_key_iv: "3c9066cdc3aa9d6ab98bb4925cfc79bd"
server_key_version: "encryptionkey@0"
I20260927 21:33:52.850178 11357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "af83bb2f411041eb9343ae4a601c6688"
format_stamp: "Formatted at 2026-09-27 21:33:52 on dist-test-slave-pqr8"
server_key: "95aaa697bf33499727f19f2e8e3d2646"
server_key_iv: "3c9066cdc3aa9d6ab98bb4925cfc79bd"
server_key_version: "encryptionkey@0"
I20260927 21:33:52.854636 11357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 21:33:52.857654 11372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:52.858891 11357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:33:52.859066 11357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "af83bb2f411041eb9343ae4a601c6688"
format_stamp: "Formatted at 2026-09-27 21:33:52 on dist-test-slave-pqr8"
server_key: "95aaa697bf33499727f19f2e8e3d2646"
server_key_iv: "3c9066cdc3aa9d6ab98bb4925cfc79bd"
server_key_version: "encryptionkey@0"
I20260927 21:33:52.859266 11357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:52.896538 11357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:52.900036 11357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:52.900281 11357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:52.920276 11357 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:33897
I20260927 21:33:52.920301 11433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:33897 every 8 connection(s)
I20260927 21:33:52.921501 11357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:33:52.924818 11434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:52.931432 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 11357
I20260927 21:33:52.931622 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:33:52.932368 3819 external_mini_cluster.cc:1428] Setting key bf808cbd951963bd0ddbb504a4170c6c
I20260927 21:33:52.935209 11434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Bootstrap starting.
I20260927 21:33:52.938409 11434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:52.939707 11434 log.cc:824] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:52.942291 11434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: No bootstrap required, opened a new log
Sep 27 21:33:52 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:52.951033 11434 raft_consensus.cc:348] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83bb2f411041eb9343ae4a601c6688" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33897 } }
I20260927 21:33:52.951396 11434 raft_consensus.cc:374] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:52.951530 11434 raft_consensus.cc:729] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af83bb2f411041eb9343ae4a601c6688, State: Initialized, Role: FOLLOWER
I20260927 21:33:52.952081 11434 consensus_queue.cc:260] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83bb2f411041eb9343ae4a601c6688" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33897 } }
I20260927 21:33:52.952359 11434 raft_consensus.cc:388] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:33:52.952500 11434 raft_consensus.cc:482] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:33:52.952666 11434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:52.954946 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:52.934243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.3.186.254:33897)
0927 21:33:52.934952 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:52.934962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:52.934997 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:33:52.936605 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:52.936625 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:52.936953 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:52.937223 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:52.939011 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:52.940834 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:52.942083 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:52.943073 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:52.943240 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:52.946776 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:52.946787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:52.946803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:52.946863 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:52.950692 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:52.951323 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:52.951330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:52.951336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:52.951600 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:52.952044 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:52.952050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:52.952055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:52.952722 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:52.953017 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:52.953318 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:33:52.953721 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":348,"threads_started":1}
I20260927 21:33:52.955564 11434 raft_consensus.cc:504] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83bb2f411041eb9343ae4a601c6688" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33897 } }
I20260927 21:33:52.955991 11434 leader_election.cc:302] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af83bb2f411041eb9343ae4a601c6688; no voters:
I20260927 21:33:52.956529 11434 leader_election.cc:288] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:33:52.956670 11439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:52.957310 11439 raft_consensus.cc:686] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [term 1 LEADER]: Becoming Leader. State: Replica: af83bb2f411041eb9343ae4a601c6688, State: Running, Role: LEADER
I20260927 21:33:52.958426 11439 consensus_queue.cc:237] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83bb2f411041eb9343ae4a601c6688" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33897 } }
I20260927 21:33:52.959424 11434 sys_catalog.cc:564] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:33:52.960902 11440 sys_catalog.cc:455] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af83bb2f411041eb9343ae4a601c6688. Latest consensus state: current_term: 1 leader_uuid: "af83bb2f411041eb9343ae4a601c6688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83bb2f411041eb9343ae4a601c6688" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33897 } } }
I20260927 21:33:52.961336 11440 sys_catalog.cc:458] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:52.961483 11441 sys_catalog.cc:455] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af83bb2f411041eb9343ae4a601c6688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83bb2f411041eb9343ae4a601c6688" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33897 } } }
I20260927 21:33:52.961808 11441 sys_catalog.cc:458] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688 [sys.catalog]: This master's current role is: LEADER
I20260927 21:33:52.969771 11444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:33:52.974339 11444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:33:52.982040 11444 catalog_manager.cc:1347] Generated new cluster ID: 35d3b282246c4281949531484a61d403
I20260927 21:33:52.982132 11444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:33:53.005602 11444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:33:53.007238 11444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:33:53.033631 11444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Generated new TSK 0
I20260927 21:33:53.034444 11444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab.
Entry for principal oryx/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab.
I20260927 21:33:53.085031 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:33897
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:53.387051 11460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:53.387281 11460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:53.387439 11460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:53.394078 11460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:53.394158 11460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:53.394344 11460 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:53.394470 11460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:33:53.403748 11460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:53.404774 11460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:53.405965 11460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:53.418352 11465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:53.418460 11466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:53.418438 11468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:53.419154 11460 server_base.cc:1034] running on GCE node
I20260927 21:33:53.419729 11460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:53.420580 11460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:53.421777 11460 hybrid_clock.cc:648] HybridClock initialized: now 1790544833421726 us; error 26 us; skew 500 ppm
I20260927 21:33:53.422040 11460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:53 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544833, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:53.424731 11460 init.cc:377] Logged in from keytab as oryx/127.3.186.193@KRBTEST.COM (short username oryx)
I20260927 21:33:53.425969 11460 webserver.cc:460] Webserver started at http://127.3.186.193:41569/ using document root <none> and password file <none>
I20260927 21:33:53.426564 11460 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:53.426682 11460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:53.426976 11460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:53.429502 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "f02db2c5eb124719b726a49ef2ca6684"
format_stamp: "Formatted at 2026-09-27 21:33:53 on dist-test-slave-pqr8"
server_key: "0f2be81eea6a7279972ba1168549be41"
server_key_iv: "6da23fc8263f7e208f1c6ec1b8e81f81"
server_key_version: "encryptionkey@0"
I20260927 21:33:53.430281 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "f02db2c5eb124719b726a49ef2ca6684"
format_stamp: "Formatted at 2026-09-27 21:33:53 on dist-test-slave-pqr8"
server_key: "0f2be81eea6a7279972ba1168549be41"
server_key_iv: "6da23fc8263f7e208f1c6ec1b8e81f81"
server_key_version: "encryptionkey@0"
I20260927 21:33:53.434590 11460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:33:53.437834 11475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:53.438948 11460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:33:53.439105 11460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "f02db2c5eb124719b726a49ef2ca6684"
format_stamp: "Formatted at 2026-09-27 21:33:53 on dist-test-slave-pqr8"
server_key: "0f2be81eea6a7279972ba1168549be41"
server_key_iv: "6da23fc8263f7e208f1c6ec1b8e81f81"
server_key_version: "encryptionkey@0"
I20260927 21:33:53.439316 11460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:53.453351 11460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:53.456873 11460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:53.457113 11460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:53.459311 11460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:53.459450 11460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:53.459602 11460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:53.459678 11460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:53 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544833, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:53.480449 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.460092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:39445 (local address 127.3.186.254:33897)
0927 21:33:53.462178 (+ 2086us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:53.462183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:53.462204 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:33:53.463779 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:53.463785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:53.463866 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:53.463989 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:53.465776 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.466447 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.467995 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.468398 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.468522 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.472966 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:53.472976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:53.472984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:53.473032 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:53.476679 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.477558 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.477564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.477568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.477704 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.478857 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.478860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.478862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.479124 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:53.479307 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:53.479892 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 21:33:53.480099 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":1898,"threads_started":1}
I20260927 21:33:53.481319 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.460671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.3.186.193:39445)
0927 21:33:53.461482 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:53.461546 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:53.463494 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:53.464200 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:53.464208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:53.464858 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:53.465614 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.465634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.466599 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.466605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.467858 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.467869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.468614 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.468621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.468811 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.469568 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:53.469591 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:53.472730 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:53.476822 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.476831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.476849 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.477368 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.477865 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.477870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.477874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.478221 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.479248 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:53.479257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:53.479573 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 21:33:53.479801 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 21:33:53.480253 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":359,"threads_started":1}
I20260927 21:33:53.505116 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.491023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:59113 (local address 127.3.186.254:33897)
0927 21:33:53.491206 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:53.491214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:53.492620 (+ 1406us) server_negotiation.cc:407] Connection header received
0927 21:33:53.492675 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:53.492681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:53.492759 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:53.492847 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:53.493862 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.494854 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.496777 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.497406 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.497559 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.499709 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:53.499714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:53.499717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:53.499748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:53.502564 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.503482 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.503485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.503487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.503587 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.503978 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.503981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.503983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.504200 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:53.504295 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:53.504779 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 21:33:53.504916 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:33:53.505931 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.491089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.3.186.193:59113)
0927 21:33:53.492320 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:53.492343 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:53.492498 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:53.493003 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:53.493006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:53.493361 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:53.493732 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.493741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.494987 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.494992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.496647 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.496657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.497623 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.497629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.497796 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.498446 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:53.498461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:53.499495 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:53.502691 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.502698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.502705 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.503367 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.503689 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.503697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.503703 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.503871 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.504305 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:53.504312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:53.504540 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 21:33:53.504719 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 21:33:53.505112 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":717,"threads_started":1}
I20260927 21:33:53.514902 11460 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:41143
I20260927 21:33:53.514936 11595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:41143 every 8 connection(s)
I20260927 21:33:53.515902 11460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:33:53.517544 11596 heartbeater.cc:347] Connected to a master server at 127.3.186.254:33897
I20260927 21:33:53.517709 11596 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:53.518203 11596 heartbeater.cc:511] Master 127.3.186.254:33897 requested a full tablet report, sending...
I20260927 21:33:53.519856 11389 ts_manager.cc:194] Registered new tserver with Master: f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143)
I20260927 21:33:53.521201 11389 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:39445
I20260927 21:33:53.522512 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 11460
I20260927 21:33:53.522702 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:33:53.523273 3819 external_mini_cluster.cc:1428] Setting key 2501c234c0405853bd018b3caf63946b
WARNING: no policy specified for oryx/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab.
Entry for principal oryx/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab.
I20260927 21:33:53.562655 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:33897
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:53.858400 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:53.858626 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:53.858711 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:53.865654 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:53.865736 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:53.865809 11600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:53.865882 11600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:33:53.874732 11600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:53.875712 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:53.876905 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:53.889985 11605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:53.889997 11608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:53.890009 11606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:53.890792 11600 server_base.cc:1034] running on GCE node
I20260927 21:33:53.891327 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:53.892143 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:53.893383 11600 hybrid_clock.cc:648] HybridClock initialized: now 1790544833893320 us; error 57 us; skew 500 ppm
I20260927 21:33:53.893692 11600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:53 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544833, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:53.896700 11600 init.cc:377] Logged in from keytab as oryx/127.3.186.194@KRBTEST.COM (short username oryx)
I20260927 21:33:53.897998 11600 webserver.cc:460] Webserver started at http://127.3.186.194:38057/ using document root <none> and password file <none>
I20260927 21:33:53.898625 11600 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:53.898748 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:53.899055 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:53.901688 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "61d5bcd0e2344759a6398ed4a3590848"
format_stamp: "Formatted at 2026-09-27 21:33:53 on dist-test-slave-pqr8"
server_key: "5e5468c278e5fcbc7d34bf2135ec7c64"
server_key_iv: "d2e8377622094918a0503f8eaf455137"
server_key_version: "encryptionkey@0"
I20260927 21:33:53.902418 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "61d5bcd0e2344759a6398ed4a3590848"
format_stamp: "Formatted at 2026-09-27 21:33:53 on dist-test-slave-pqr8"
server_key: "5e5468c278e5fcbc7d34bf2135ec7c64"
server_key_iv: "d2e8377622094918a0503f8eaf455137"
server_key_version: "encryptionkey@0"
I20260927 21:33:53.906984 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:33:53.910192 11615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:53.911370 11600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:53.911550 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "61d5bcd0e2344759a6398ed4a3590848"
format_stamp: "Formatted at 2026-09-27 21:33:53 on dist-test-slave-pqr8"
server_key: "5e5468c278e5fcbc7d34bf2135ec7c64"
server_key_iv: "d2e8377622094918a0503f8eaf455137"
server_key_version: "encryptionkey@0"
I20260927 21:33:53.911780 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:53.922587 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:53.926256 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:53.926493 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:53.929031 11600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:53.929171 11600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:33:53.929334 11600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:53.929462 11600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:33:53 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544833, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:53.950197 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.929570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:44611 (local address 127.3.186.254:33897)
0927 21:33:53.929716 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:53.929738 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:53.931190 (+ 1452us) server_negotiation.cc:407] Connection header received
0927 21:33:53.933824 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:53.933830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:53.933903 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:53.933993 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:53.935761 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.936882 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.939716 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.940282 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.940382 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.944926 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:53.944937 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:53.944941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:53.944977 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:53.947504 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.948146 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.948152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.948155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.948267 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.948644 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.948649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.948652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.948894 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:53.949042 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:53.949669 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 21:33:53.949864 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:33:53.951035 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.930167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.3.186.194:44611)
0927 21:33:53.931044 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:53.931098 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:53.933507 (+ 2409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:53.934397 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:53.934406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:53.934985 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:53.935591 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.935610 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.937288 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.937292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.939517 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.939539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.940759 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.940774 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.941143 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.941943 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:53.941977 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:53.944721 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:53.947633 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.947642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.947654 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.948029 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.948367 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.948373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.948377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.948535 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.949023 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:53.949029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:53.949309 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 21:33:53.949585 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 21:33:53.950056 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":380,"threads_started":1}
I20260927 21:33:53.969702 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.955951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:57221 (local address 127.3.186.254:33897)
0927 21:33:53.956126 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:53.956133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:53.956898 (+ 765us) server_negotiation.cc:407] Connection header received
0927 21:33:53.957084 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:53.957089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:53.957151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:53.957239 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:53.958600 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.959576 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.961629 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.961997 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.962125 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.964397 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:53.964401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:53.964404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:53.964430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:53.967170 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.967924 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.967932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.967937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.968051 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:53.968368 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:53.968374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:53.968377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:53.968646 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:53.968786 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:53.969129 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 21:33:53.969310 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:33:53.969813 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:53.956009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.3.186.194:57221)
0927 21:33:53.956810 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:53.956824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:53.956962 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:53.957257 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:53.957261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:53.957842 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:53.958398 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.958416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.959702 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.959709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.961456 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:53.961471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:53.962172 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:53.962180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:53.962330 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:53.962907 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:53.962936 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:53.964214 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:53.967284 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.967291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.967295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.967812 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.968155 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:53.968160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:53.968164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:53.968274 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:53.968756 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:53.968760 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:53.968917 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 21:33:53.969049 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:33:53.969313 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":452,"threads_started":1}
I20260927 21:33:53.988648 11600 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:41307
I20260927 21:33:53.988667 11734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:41307 every 8 connection(s)
I20260927 21:33:53.989776 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:33:53.991549 11735 heartbeater.cc:347] Connected to a master server at 127.3.186.254:33897
I20260927 21:33:53.991722 11735 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:53.992223 11735 heartbeater.cc:511] Master 127.3.186.254:33897 requested a full tablet report, sending...
I20260927 21:33:53.993310 11389 ts_manager.cc:194] Registered new tserver with Master: 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307)
I20260927 21:33:53.994084 11389 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:44611
I20260927 21:33:53.998044 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 11600
I20260927 21:33:53.998191 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:33:53.998597 3819 external_mini_cluster.cc:1428] Setting key 747e42e852cfd696571e950b1fc6564e
WARNING: no policy specified for oryx/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab.
Entry for principal oryx/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab.
I20260927 21:33:54.038599 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:33897
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:54.351415 11739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:54.351636 11739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:54.351720 11739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:54.358546 11739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:54.358655 11739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:54.358825 11739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:33:54.358968 11739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:33:54.368239 11739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.186.254:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:54.369289 11739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:54.370539 11739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:54.382397 11747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:54.382493 11744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:54.382469 11745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:54.383503 11739 server_base.cc:1034] running on GCE node
I20260927 21:33:54.384087 11739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:54.384928 11739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:54.386165 11739 hybrid_clock.cc:648] HybridClock initialized: now 1790544834386130 us; error 57 us; skew 500 ppm
I20260927 21:33:54.386435 11739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:33:54 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544834, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:33:54.389668 11739 init.cc:377] Logged in from keytab as oryx/127.3.186.195@KRBTEST.COM (short username oryx)
I20260927 21:33:54.390915 11739 webserver.cc:460] Webserver started at http://127.3.186.195:34651/ using document root <none> and password file <none>
I20260927 21:33:54.391548 11739 fs_manager.cc:359] Metadata directory not provided
I20260927 21:33:54.391674 11739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:33:54.391991 11739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:33:54.394740 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
format_stamp: "Formatted at 2026-09-27 21:33:54 on dist-test-slave-pqr8"
server_key: "c8f8172d4cd23e584eab2eae0474b60c"
server_key_iv: "040362f6ad230c1beca538f4be6b3881"
server_key_version: "encryptionkey@0"
I20260927 21:33:54.395494 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
format_stamp: "Formatted at 2026-09-27 21:33:54 on dist-test-slave-pqr8"
server_key: "c8f8172d4cd23e584eab2eae0474b60c"
server_key_iv: "040362f6ad230c1beca538f4be6b3881"
server_key_version: "encryptionkey@0"
I20260927 21:33:54.400178 11739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:33:54.403259 11754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:54.404490 11739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:54.404652 11739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
format_stamp: "Formatted at 2026-09-27 21:33:54 on dist-test-slave-pqr8"
server_key: "c8f8172d4cd23e584eab2eae0474b60c"
server_key_iv: "040362f6ad230c1beca538f4be6b3881"
server_key_version: "encryptionkey@0"
I20260927 21:33:54.404845 11739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:33:54.442675 11739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:33:54.446681 11739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:33:54.446977 11739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:33:54.449255 11739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:33:54.449397 11739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:33:54.449582 11739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:33:54.449662 11739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 27 21:33:54 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544834, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:54.471654 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:54.449943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:60941 (local address 127.3.186.254:33897)
0927 21:33:54.450121 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:54.450127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:54.451424 (+ 1297us) server_negotiation.cc:407] Connection header received
0927 21:33:54.453297 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:54.453304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:54.453429 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:54.453610 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:54.455791 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.457052 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.458918 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.459771 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.459915 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:54.464245 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:54.464254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:54.464260 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:54.464335 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:54.468226 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:54.468855 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:54.468863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:54.468869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:54.469045 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:54.469531 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:54.469538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:54.469544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:54.470019 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:54.470189 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:54.470827 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 21:33:54.471126 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:33:54.472147 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:54.450461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.3.186.195:60941)
0927 21:33:54.451240 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:54.451285 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:54.453044 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:54.453855 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:54.453864 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:54.454645 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:54.455578 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:54.455612 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.457212 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.457218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:54.458759 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:54.458770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.460056 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.460064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:54.460286 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:54.461015 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:54.461040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:54.463980 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:54.468363 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:54.468372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:54.468384 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:54.468742 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:54.469191 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:54.469196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:54.469200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:54.469379 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:54.470175 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:54.470185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:54.470476 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 21:33:54.470690 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 21:33:54.471128 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":369,"threads_started":1}
I20260927 21:33:54.491165 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:54.477439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:52637 (local address 127.3.186.254:33897)
0927 21:33:54.477641 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:54.477649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:54.478289 (+ 640us) server_negotiation.cc:407] Connection header received
0927 21:33:54.478478 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:54.478481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:54.478531 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:54.478636 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:54.479929 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.480914 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.482913 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.483312 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.483443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:54.485030 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:54.485035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:54.485040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:54.485077 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:54.488193 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:54.489087 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:54.489091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:54.489093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:54.489190 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:54.489705 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:54.489709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:54.489711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:54.489997 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:54.490138 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:54.490605 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 21:33:54.490796 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:54.491307 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:54.477444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.3.186.195:52637)
0927 21:33:54.478173 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:54.478196 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:54.478362 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:54.478657 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:54.478662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:54.479332 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:54.479793 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:54.479804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.481037 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.481042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:54.482768 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:54.482779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.483488 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.483495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:54.483669 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:54.484236 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:54.484251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:54.484861 (+ 610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:54.488350 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:54.488355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:54.488359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:54.488928 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:54.489336 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:54.489342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:54.489347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:54.489567 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:54.490208 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:54.490211 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:54.490341 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 21:33:54.490485 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:33:54.490796 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":413,"threads_started":1}
I20260927 21:33:54.500208 11739 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:33175
I20260927 21:33:54.500247 11873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:33175 every 8 connection(s)
I20260927 21:33:54.501269 11739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:33:54.503054 11874 heartbeater.cc:347] Connected to a master server at 127.3.186.254:33897
I20260927 21:33:54.503275 11874 heartbeater.cc:464] Registering TS with master...
I20260927 21:33:54.503823 11874 heartbeater.cc:511] Master 127.3.186.254:33897 requested a full tablet report, sending...
I20260927 21:33:54.504941 11389 ts_manager.cc:194] Registered new tserver with Master: f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:54.505712 11389 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:60941
I20260927 21:33:54.506157 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 11739
I20260927 21:33:54.506328 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:33:54.506714 3819 external_mini_cluster.cc:1428] Setting key e2d23d0766f81472648104842e5e9c26
I20260927 21:33:54.509387 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:33:54.523530 11596 heartbeater.cc:503] Master 127.3.186.254:33897 was elected leader, sending a full tablet report...
Sep 27 21:33:54 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:33:54 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:33:54 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:33:54.996114 11735 heartbeater.cc:503] Master 127.3.186.254:33897 was elected leader, sending a full tablet report...
I20260927 21:33:55.507885 11874 heartbeater.cc:503] Master 127.3.186.254:33897 was elected leader, sending a full tablet report...
W20260927 21:33:55.586315 11882 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.038s user 0.003s sys 0.009s
W20260927 21:33:55.587386 11484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:54.543895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.3.186.254:33897)
0927 21:33:54.544031 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:54.544035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:54.544350 (+ 315us) server_negotiation.cc:407] Connection header received
0927 21:33:54.544465 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:54.544468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:54.544520 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:54.544612 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:54.545817 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.546483 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.547664 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:54.548045 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:54.548128 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.587172 (+1039044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34138: BlockingRecv error: recv got EOF from 127.0.0.1:34138 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 21:33:55 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:55.606452 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.593758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.3.186.254:33897)
0927 21:33:55.593996 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.594003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.594362 (+ 359us) server_negotiation.cc:407] Connection header received
0927 21:33:55.594621 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.594626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.594751 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.594831 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:55.595936 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.596627 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.597910 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.598401 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.598544 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.602053 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:55.602058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:55.602061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:55.602089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:55.604285 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:55.604867 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:55.604871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:55.604873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:55.604981 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:55.605293 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:55.605296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:55.605298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:55.605589 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:55.605797 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.606012 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.606150 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:55.610204 11389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:55.617156 11389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:55.638998 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.627680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.3.186.195:33175)
0927 21:33:55.628341 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.628350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.628382 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:33:55.628749 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.628762 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.630841 (+ 2079us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.631058 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.632305 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.633738 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.636475 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.637931 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.638064 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.638262 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.638560 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.638720 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":714,"threads_started":1}
I20260927 21:33:55.639354 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.628635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41143 (local address 127.0.0.1:45454)
0927 21:33:55.632195 (+ 3560us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.632211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.632313 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.633126 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.633129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.633146 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.633570 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.633582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.634833 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.634839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.636848 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.636860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.638455 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.638464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.638636 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.638669 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.638749 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.638870 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3454,"mutex_wait_us":55,"thread_start_us":3287,"threads_started":1}
I20260927 21:33:55.640005 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.628541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.3.186.194:41307)
0927 21:33:55.629667 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.629680 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.629714 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:33:55.629825 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.629835 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.630142 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.630370 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.631245 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.633206 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.637054 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.638413 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.639207 (+ 794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.639420 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.639507 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.639720 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":808,"threads_started":1}
I20260927 21:33:55.639524 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.628128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:41307 (local address 127.0.0.1:59904)
0927 21:33:55.628965 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.628981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.629122 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.630412 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.630423 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.630465 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.630989 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.631004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.633377 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.633383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.636317 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.636331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.638671 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.638678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.638830 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.638863 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.638927 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.639074 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":187,"thread_start_us":587,"threads_started":1}
I20260927 21:33:55.640301 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.629069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.3.186.193:41143)
0927 21:33:55.631700 (+ 2631us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.631711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.632424 (+ 713us) server_negotiation.cc:407] Connection header received
0927 21:33:55.632537 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.632544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.632782 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.632942 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.633767 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.634707 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.637013 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.637979 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.638136 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.638321 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.639710 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.639895 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":2285,"threads_started":1}
I20260927 21:33:55.639091 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.627502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:33175 (local address 127.0.0.1:53874)
0927 21:33:55.628272 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.628312 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.628446 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.631593 (+ 3147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.631598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.631636 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.632130 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.632142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.633862 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.633867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.636317 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.636331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.638140 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.638146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.638310 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.638360 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.638492 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.638667 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":150,"thread_start_us":381,"threads_started":1}
I20260927 21:33:55.643431 11526 tablet_service.cc:1467] Processing CreateTablet for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd (DEFAULT_TABLE table=test-table [id=d54fdc62a80044d8a0ec75039e6747a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:55.643626 11804 tablet_service.cc:1467] Processing CreateTablet for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd (DEFAULT_TABLE table=test-table [id=d54fdc62a80044d8a0ec75039e6747a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:55.643617 11670 tablet_service.cc:1467] Processing CreateTablet for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd (DEFAULT_TABLE table=test-table [id=d54fdc62a80044d8a0ec75039e6747a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:55.644694 11526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:55.644834 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:55.644834 11804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:55.658257 11901 tablet_bootstrap.cc:492] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Bootstrap starting.
I20260927 21:33:55.658706 11900 tablet_bootstrap.cc:492] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Bootstrap starting.
I20260927 21:33:55.661290 11899 tablet_bootstrap.cc:492] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Bootstrap starting.
I20260927 21:33:55.662900 11900 tablet_bootstrap.cc:654] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:55.664062 11900 log.cc:824] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:55.664484 11899 tablet_bootstrap.cc:654] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:55.664757 11901 tablet_bootstrap.cc:654] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:55.665757 11901 log.cc:824] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:55.665894 11899 log.cc:824] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Log is configured to *not* fsync() on all Append() calls
I20260927 21:33:55.668004 11900 tablet_bootstrap.cc:492] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: No bootstrap required, opened a new log
I20260927 21:33:55.668285 11900 ts_tablet_manager.cc:1397] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260927 21:33:55.668588 11899 tablet_bootstrap.cc:492] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: No bootstrap required, opened a new log
I20260927 21:33:55.668655 11901 tablet_bootstrap.cc:492] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: No bootstrap required, opened a new log
I20260927 21:33:55.668819 11899 ts_tablet_manager.cc:1397] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:33:55.668852 11901 ts_tablet_manager.cc:1397] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 21:33:55.675858 11901 raft_consensus.cc:348] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.676167 11901 raft_consensus.cc:374] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:55.676276 11901 raft_consensus.cc:729] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Initialized, Role: FOLLOWER
I20260927 21:33:55.676808 11901 consensus_queue.cc:260] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.678506 11899 raft_consensus.cc:348] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.678506 11900 raft_consensus.cc:348] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.678881 11900 raft_consensus.cc:374] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:55.678881 11899 raft_consensus.cc:374] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:55.678977 11900 raft_consensus.cc:729] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45a39838c5b47ea8e77a7b5169e85ee, State: Initialized, Role: FOLLOWER
I20260927 21:33:55.678980 11899 raft_consensus.cc:729] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02db2c5eb124719b726a49ef2ca6684, State: Initialized, Role: FOLLOWER
I20260927 21:33:55.679530 11899 consensus_queue.cc:260] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.679559 11900 consensus_queue.cc:260] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.682533 11900 ts_tablet_manager.cc:1428] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260927 21:33:55.684279 11899 ts_tablet_manager.cc:1428] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 21:33:55.687059 11901 ts_tablet_manager.cc:1428] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Time spent starting tablet: real 0.018s user 0.006s sys 0.008s
W20260927 21:33:55.742180 11736 tablet.cc:2369] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:33:55.753656 11875 tablet.cc:2369] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:33:55.768384 11597 tablet.cc:2369] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:33:55.946827 11905 raft_consensus.cc:482] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:55.947148 11905 raft_consensus.cc:504] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.948655 11905 leader_election.cc:288] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175), f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143)
I20260927 21:33:55.954485 11906 raft_consensus.cc:482] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:55.954752 11906 raft_consensus.cc:504] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.955760 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.949965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:49961 (local address 127.3.186.193:41143)
0927 21:33:55.950144 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.950151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.950517 (+ 366us) server_negotiation.cc:407] Connection header received
0927 21:33:55.950740 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.950743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.950815 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.950898 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.951361 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.952010 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.953837 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.954778 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.954975 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.955036 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.955390 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.955522 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:33:55.955768 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.949131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:59245 (local address 127.3.186.195:33175)
0927 21:33:55.949344 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.949352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.949867 (+ 515us) server_negotiation.cc:407] Connection header received
0927 21:33:55.950077 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.950089 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.950152 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.950250 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.950833 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.951499 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.953349 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.954152 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.954305 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.954375 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.955434 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.955544 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:33:55.955837 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.949780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41143 (local address 127.3.186.194:49961)
0927 21:33:55.950416 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.950429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.950584 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.950909 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.950913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.950932 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.951225 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.951233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.952141 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.952146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.953693 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.953704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.954992 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.954998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.955209 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.955259 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.955352 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.955478 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":29,"thread_start_us":358,"threads_started":1}
I20260927 21:33:55.956336 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.949023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:33175 (local address 127.3.186.194:59245)
0927 21:33:55.949781 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.949800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.949942 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.950275 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.950280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.950309 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.950697 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.950706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.951627 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.951633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.953170 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.953181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.954343 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.954349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.955274 (+ 925us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.955309 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.955933 (+ 624us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.956061 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":62,"thread_start_us":368,"threads_started":1}
I20260927 21:33:55.956544 11906 leader_election.cc:288] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175), 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307)
I20260927 21:33:55.956811 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608472bfc2f94a3dbf4606f4c3b4d2fd" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684" is_pre_election: true
I20260927 21:33:55.957321 11551 raft_consensus.cc:2454] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 0.
I20260927 21:33:55.958079 11619 leader_election.cc:302] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f02db2c5eb124719b726a49ef2ca6684; no voters:
I20260927 21:33:55.958000 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608472bfc2f94a3dbf4606f4c3b4d2fd" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" is_pre_election: true
I20260927 21:33:55.958484 11905 raft_consensus.cc:2781] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:55.958506 11829 raft_consensus.cc:2454] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 0.
I20260927 21:33:55.958616 11905 raft_consensus.cc:482] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:55.958690 11905 raft_consensus.cc:3037] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:55.962611 11905 raft_consensus.cc:504] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.963338 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.957017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:39963 (local address 127.3.186.195:33175)
0927 21:33:55.957154 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.957157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.957870 (+ 713us) server_negotiation.cc:407] Connection header received
0927 21:33:55.958150 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.958155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.958226 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.958322 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.958743 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.959745 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.961561 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.962359 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.962573 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.962636 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.962989 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.963096 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:33:55.963428 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.957155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:33175 (local address 127.3.186.193:39963)
0927 21:33:55.957784 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.957799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.957959 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.958353 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.958357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.958380 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.958621 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.958627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.959858 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.959865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.961381 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.961391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.962608 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.962614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.962771 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.962804 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.962950 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.963062 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":2134,"threads_started":1}
I20260927 21:33:55.963721 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608472bfc2f94a3dbf4606f4c3b4d2fd" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
I20260927 21:33:55.963912 11829 raft_consensus.cc:3037] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:55.964087 11828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608472bfc2f94a3dbf4606f4c3b4d2fd" candidate_uuid: "f02db2c5eb124719b726a49ef2ca6684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" is_pre_election: true
I20260927 21:33:55.967928 11829 raft_consensus.cc:2454] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 1.
I20260927 21:33:55.968408 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.959102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:41307 (local address 127.3.186.193:48929)
0927 21:33:55.963737 (+ 4635us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:55.963752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:55.963873 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:55.964333 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:55.964338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:55.964360 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:55.964529 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.964539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.965639 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.965642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.966687 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:55.966696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.967714 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.967720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:55.967905 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.967925 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 21:33:55.968021 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:33:55.968158 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4551,"threads_started":1}
I20260927 21:33:55.968387 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.957694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:48929 (local address 127.3.186.194:41307)
0927 21:33:55.958988 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.958994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.963828 (+ 4834us) server_negotiation.cc:407] Connection header received
0927 21:33:55.964006 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.964011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.964095 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.964156 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:55.964671 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.965454 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.966827 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.967560 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:55.967721 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:55.967788 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:55.968042 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:33:55.968165 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148}
I20260927 21:33:55.968413 11619 leader_election.cc:302] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f45a39838c5b47ea8e77a7b5169e85ee; no voters:
I20260927 21:33:55.969084 11912 raft_consensus.cc:2781] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:55.969409 11905 leader_election.cc:288] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 election: Requested vote from peers f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175), f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143)
I20260927 21:33:55.969862 11912 raft_consensus.cc:686] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 1 LEADER]: Becoming Leader. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Running, Role: LEADER
I20260927 21:33:55.970285 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608472bfc2f94a3dbf4606f4c3b4d2fd" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684"
I20260927 21:33:55.970494 11551 raft_consensus.cc:3037] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:55.970429 11912 consensus_queue.cc:237] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } }
I20260927 21:33:55.971232 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608472bfc2f94a3dbf4606f4c3b4d2fd" candidate_uuid: "f02db2c5eb124719b726a49ef2ca6684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d5bcd0e2344759a6398ed4a3590848" is_pre_election: true
I20260927 21:33:55.972218 11477 leader_election.cc:302] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f02db2c5eb124719b726a49ef2ca6684; no voters: 61d5bcd0e2344759a6398ed4a3590848, f45a39838c5b47ea8e77a7b5169e85ee
I20260927 21:33:55.973387 11551 raft_consensus.cc:2454] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 1.
I20260927 21:33:55.973631 11906 raft_consensus.cc:2726] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 21:33:55.975450 11388 catalog_manager.cc:5495] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "61d5bcd0e2344759a6398ed4a3590848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } health_report { overall_health: HEALTHY } } }
I20260927 21:33:56.002110 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:55.996956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.3.186.194:41307)
0927 21:33:55.997114 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:55.997122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:55.997139 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:33:55.997367 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:55.997372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:55.997448 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:55.997603 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:55.997953 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:55.998753 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.000140 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.000551 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.000681 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.001097 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.001568 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.001710 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.001784 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.001851 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:33:56.008848 11551 raft_consensus.cc:1261] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Refusing update from remote peer 61d5bcd0e2344759a6398ed4a3590848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:56.008865 11829 raft_consensus.cc:1261] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Refusing update from remote peer 61d5bcd0e2344759a6398ed4a3590848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:56.009606 11912 consensus_queue.cc:1035] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:56.009855 11913 consensus_queue.cc:1035] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:56.019764 11916 mvcc.cc:205] Tried to move back new op lower bound from 7334071648284749824 to 7334071648139808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334071648284749824}]
I20260927 21:33:56.020613 11917 mvcc.cc:205] Tried to move back new op lower bound from 7334071648284749824 to 7334071648139808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334071648284749824}]
I20260927 21:33:56.024015 11918 mvcc.cc:205] Tried to move back new op lower bound from 7334071648284749824 to 7334071648139808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334071648284749824 or (T in {7334071648284749824})}]
I20260927 21:33:56.031268 11387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:56.031510 11387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:56.034543 11387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Sending DeleteTablet for 3 replicas of tablet 608472bfc2f94a3dbf4606f4c3b4d2fd
I20260927 21:33:56.035466 11804 tablet_service.cc:1514] Processing DeleteTablet for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:56 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:53874
I20260927 21:33:56.035558 11526 tablet_service.cc:1514] Processing DeleteTablet for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:56 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45454
I20260927 21:33:56.035565 11670 tablet_service.cc:1514] Processing DeleteTablet for tablet 608472bfc2f94a3dbf4606f4c3b4d2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:56 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59904
I20260927 21:33:56.040037 11388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:56.040866 11388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:56.047974 11926 tablet_replica.cc:330] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: stopping tablet replica
I20260927 21:33:56.048352 11926 raft_consensus.cc:2229] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:56.050104 11926 raft_consensus.cc:2258] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:56.050554 11803 tablet_service.cc:1467] Processing CreateTablet for tablet b7fbe916deda42fdb55e557e791bf480 (DEFAULT_TABLE table=test-table [id=20daf49f398e4bf4be67b0c952773925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:56.050714 11525 tablet_service.cc:1467] Processing CreateTablet for tablet b7fbe916deda42fdb55e557e791bf480 (DEFAULT_TABLE table=test-table [id=20daf49f398e4bf4be67b0c952773925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:56.051067 11803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fbe916deda42fdb55e557e791bf480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.051051 11669 tablet_service.cc:1467] Processing CreateTablet for tablet b7fbe916deda42fdb55e557e791bf480 (DEFAULT_TABLE table=test-table [id=20daf49f398e4bf4be67b0c952773925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:56.051201 11525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fbe916deda42fdb55e557e791bf480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.051539 11669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fbe916deda42fdb55e557e791bf480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.051877 11925 tablet_replica.cc:330] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: stopping tablet replica
I20260927 21:33:56.052245 11925 raft_consensus.cc:2229] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:56.052664 11925 raft_consensus.cc:2258] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:56.056079 11925 ts_tablet_manager.cc:1905] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:56.057333 11899 tablet_bootstrap.cc:492] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Bootstrap starting.
I20260927 21:33:56.057333 11900 tablet_bootstrap.cc:492] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Bootstrap starting.
I20260927 21:33:56.058889 11899 tablet_bootstrap.cc:654] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.058893 11900 tablet_bootstrap.cc:654] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.061810 11901 tablet_bootstrap.cc:492] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Bootstrap starting.
I20260927 21:33:56.063361 11901 tablet_bootstrap.cc:654] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.069293 11925 ts_tablet_manager.cc:1918] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:56.069438 11925 log.cc:1196] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/608472bfc2f94a3dbf4606f4c3b4d2fd
I20260927 21:33:56.071259 11927 tablet_replica.cc:330] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: stopping tablet replica
I20260927 21:33:56.071673 11927 raft_consensus.cc:2229] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:56.072630 11925 ts_tablet_manager.cc:1939] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f45a39838c5b47ea8e77a7b5169e85ee: Deleting consensus metadata
I20260927 21:33:56.072250 11901 tablet_bootstrap.cc:492] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: No bootstrap required, opened a new log
I20260927 21:33:56.073339 11901 ts_tablet_manager.cc:1397] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260927 21:33:56.073969 11900 tablet_bootstrap.cc:492] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: No bootstrap required, opened a new log
I20260927 21:33:56.074097 11900 ts_tablet_manager.cc:1397] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20260927 21:33:56.074179 11901 raft_consensus.cc:348] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.074398 11901 raft_consensus.cc:374] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.074489 11901 raft_consensus.cc:729] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.074694 11376 catalog_manager.cc:4841] TS f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175): tablet 608472bfc2f94a3dbf4606f4c3b4d2fd (table test-table [id=d54fdc62a80044d8a0ec75039e6747a0]) successfully deleted
I20260927 21:33:56.074759 11901 consensus_queue.cc:260] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.074865 11900 raft_consensus.cc:348] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.075104 11900 raft_consensus.cc:374] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.075197 11900 raft_consensus.cc:729] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45a39838c5b47ea8e77a7b5169e85ee, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.075455 11900 consensus_queue.cc:260] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.077368 11901 ts_tablet_manager.cc:1428] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 21:33:56.077812 11900 ts_tablet_manager.cc:1428] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:33:56.080132 11926 ts_tablet_manager.cc:1905] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:56.084683 11926 ts_tablet_manager.cc:1918] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:56.084841 11926 log.cc:1196] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/608472bfc2f94a3dbf4606f4c3b4d2fd
I20260927 21:33:56.084940 11927 raft_consensus.cc:2258] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:56.085367 11926 ts_tablet_manager.cc:1939] T 608472bfc2f94a3dbf4606f4c3b4d2fd P f02db2c5eb124719b726a49ef2ca6684: Deleting consensus metadata
I20260927 21:33:56.087224 11376 catalog_manager.cc:4841] TS f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143): tablet 608472bfc2f94a3dbf4606f4c3b4d2fd (table test-table [id=d54fdc62a80044d8a0ec75039e6747a0]) successfully deleted
I20260927 21:33:56.087949 11899 tablet_bootstrap.cc:492] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: No bootstrap required, opened a new log
I20260927 21:33:56.088038 11927 ts_tablet_manager.cc:1905] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:56.088116 11899 ts_tablet_manager.cc:1397] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.007s
I20260927 21:33:56.088886 11899 raft_consensus.cc:348] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.089095 11899 raft_consensus.cc:374] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.089184 11899 raft_consensus.cc:729] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02db2c5eb124719b726a49ef2ca6684, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.089409 11899 consensus_queue.cc:260] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.090116 11899 ts_tablet_manager.cc:1428] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:56.094424 11927 ts_tablet_manager.cc:1918] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:56.094587 11927 log.cc:1196] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/608472bfc2f94a3dbf4606f4c3b4d2fd
I20260927 21:33:56.095165 11927 ts_tablet_manager.cc:1939] T 608472bfc2f94a3dbf4606f4c3b4d2fd P 61d5bcd0e2344759a6398ed4a3590848: Deleting consensus metadata
I20260927 21:33:56.096374 11374 catalog_manager.cc:4841] TS 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307): tablet 608472bfc2f94a3dbf4606f4c3b4d2fd (table test-table [id=d54fdc62a80044d8a0ec75039e6747a0]) successfully deleted
I20260927 21:33:56.331573 11907 raft_consensus.cc:482] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:56.331837 11907 raft_consensus.cc:504] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.333365 11907 leader_election.cc:288] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143), 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307)
I20260927 21:33:56.341197 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.333769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:57341 (local address 127.3.186.194:41307)
0927 21:33:56.334354 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.334360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.334454 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:33:56.334735 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.334744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.334823 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.334884 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:56.335761 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.336487 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.339283 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.340111 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.340291 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.340368 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.340730 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.340865 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260927 21:33:56.341390 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.334521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:36521 (local address 127.3.186.193:41143)
0927 21:33:56.334692 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.334698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.334718 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:33:56.334792 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.334798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.334874 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.334977 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:33:56.335625 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.336685 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.339216 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.340445 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.340615 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.340694 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.340992 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.341133 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:33:56.341784 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.333818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41143 (local address 127.3.186.195:36521)
0927 21:33:56.334563 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:56.334578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:56.334667 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:56.335014 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:56.335021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:56.335055 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:56.335423 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.335435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.336829 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.336834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.339019 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.339050 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.340676 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.340685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.340830 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.340858 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:33:56.340940 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:33:56.341062 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":77,"thread_start_us":966,"threads_started":1}
I20260927 21:33:56.341971 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.333648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:41307 (local address 127.3.186.195:57341)
0927 21:33:56.334331 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:56.334359 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:56.334565 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:56.335085 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:56.335091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:56.335114 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:33:56.335586 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.335598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.336616 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.336624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.339079 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.339091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.340332 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.340338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.340502 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.340557 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 21:33:56.341408 (+ 851us) client_negotiation.cc:240] Negotiation successful
0927 21:33:56.341564 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":73,"thread_start_us":366,"threads_started":1}
I20260927 21:33:56.342522 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fbe916deda42fdb55e557e791bf480" candidate_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d5bcd0e2344759a6398ed4a3590848" is_pre_election: true
I20260927 21:33:56.342548 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fbe916deda42fdb55e557e791bf480" candidate_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684" is_pre_election: true
I20260927 21:33:56.342792 11551 raft_consensus.cc:2454] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45a39838c5b47ea8e77a7b5169e85ee in term 0.
I20260927 21:33:56.342861 11690 raft_consensus.cc:2454] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45a39838c5b47ea8e77a7b5169e85ee in term 0.
I20260927 21:33:56.343425 11756 leader_election.cc:302] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f45a39838c5b47ea8e77a7b5169e85ee; no voters:
I20260927 21:33:56.343751 11907 raft_consensus.cc:2781] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:56.343865 11907 raft_consensus.cc:482] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:56.343958 11907 raft_consensus.cc:3037] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:56.346328 11907 raft_consensus.cc:504] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.346875 11907 leader_election.cc:288] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [CANDIDATE]: Term 1 election: Requested vote from peers f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143), 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307)
I20260927 21:33:56.347313 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fbe916deda42fdb55e557e791bf480" candidate_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684"
I20260927 21:33:56.347337 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fbe916deda42fdb55e557e791bf480" candidate_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d5bcd0e2344759a6398ed4a3590848"
I20260927 21:33:56.347515 11551 raft_consensus.cc:3037] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:56.347529 11690 raft_consensus.cc:3037] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:56.350370 11551 raft_consensus.cc:2454] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45a39838c5b47ea8e77a7b5169e85ee in term 1.
I20260927 21:33:56.350703 11690 raft_consensus.cc:2454] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45a39838c5b47ea8e77a7b5169e85ee in term 1.
I20260927 21:33:56.350764 11758 leader_election.cc:302] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f02db2c5eb124719b726a49ef2ca6684, f45a39838c5b47ea8e77a7b5169e85ee; no voters:
I20260927 21:33:56.351047 11907 raft_consensus.cc:2781] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:56.351490 11907 raft_consensus.cc:686] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 LEADER]: Becoming Leader. State: Replica: f45a39838c5b47ea8e77a7b5169e85ee, State: Running, Role: LEADER
I20260927 21:33:56.352023 11907 consensus_queue.cc:237] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.355337 11388 catalog_manager.cc:5495] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee reported cstate change: term changed from 0 to 1, leader changed from <none> to f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195). New cstate: current_term: 1 leader_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } health_report { overall_health: HEALTHY } } }
Sep 27 21:33:56 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544832, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.254@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:56.456377 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.443903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.3.186.254:33897)
0927 21:33:56.444440 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.444444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.444803 (+ 359us) server_negotiation.cc:407] Connection header received
0927 21:33:56.444924 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.444927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.444991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.445080 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:56.446091 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.446781 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.447950 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.448363 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.448448 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.451545 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:56.451550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:56.451553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:56.451580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:56.454259 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:56.454819 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:56.454822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:56.454824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:56.454927 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:56.455232 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:56.455235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:56.455237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:56.455455 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:56.455598 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.455915 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.456111 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
I20260927 21:33:56.456698 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.443903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:33897 (local address 127.0.0.1:34170)
0927 21:33:56.444714 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:56.444733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:56.444837 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:56.445125 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:56.445127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:56.445783 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:33:56.445976 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.445984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.446886 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.446889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.447839 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.447849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.448493 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.448500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.448618 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.449202 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:33:56.449219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:33:56.451308 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:33:56.454368 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:56.454377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:56.454381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:56.454728 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:56.455013 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:33:56.455015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:33:56.455017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:33:56.455149 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:33:56.455576 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:33:56.455582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:33:56.455721 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:33:56.455846 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 21:33:56.456010 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"spinlock_wait_cycles":10624,"thread_start_us":535,"threads_started":1}
I20260927 21:33:56.461128 11388 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:34170:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:33:56.462132 11388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:56.468143 11525 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0a6cf9b1534a69bdab906d1f0c66f5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f142218b01874106a397050a5a1c626f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:56.468281 11670 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0a6cf9b1534a69bdab906d1f0c66f5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f142218b01874106a397050a5a1c626f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:56.468431 11804 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0a6cf9b1534a69bdab906d1f0c66f5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f142218b01874106a397050a5a1c626f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:33:56.468674 11525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0a6cf9b1534a69bdab906d1f0c66f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.468694 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0a6cf9b1534a69bdab906d1f0c66f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.468981 11804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0a6cf9b1534a69bdab906d1f0c66f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.474483 11900 tablet_bootstrap.cc:492] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: Bootstrap starting.
I20260927 21:33:56.475251 11901 tablet_bootstrap.cc:492] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: Bootstrap starting.
I20260927 21:33:56.475857 11900 tablet_bootstrap.cc:654] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.476327 11901 tablet_bootstrap.cc:654] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.476768 11899 tablet_bootstrap.cc:492] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: Bootstrap starting.
I20260927 21:33:56.477609 11900 tablet_bootstrap.cc:492] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: No bootstrap required, opened a new log
I20260927 21:33:56.477803 11900 ts_tablet_manager.cc:1397] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:33:56.478359 11899 tablet_bootstrap.cc:654] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.478405 11900 raft_consensus.cc:348] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.478665 11900 raft_consensus.cc:374] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.478744 11900 raft_consensus.cc:729] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45a39838c5b47ea8e77a7b5169e85ee, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.479045 11901 tablet_bootstrap.cc:492] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: No bootstrap required, opened a new log
I20260927 21:33:56.478972 11900 consensus_queue.cc:260] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.479403 11907 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.479566 11900 ts_tablet_manager.cc:1428] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 21:33:56.479712 11907 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.479205 11901 ts_tablet_manager.cc:1397] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260927 21:33:56.480377 11901 raft_consensus.cc:348] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.480630 11901 raft_consensus.cc:374] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.480710 11901 raft_consensus.cc:729] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.480924 11901 consensus_queue.cc:260] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.481622 11912 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.481848 11912 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.482227 11901 ts_tablet_manager.cc:1428] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:33:56.483659 11899 tablet_bootstrap.cc:492] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: No bootstrap required, opened a new log
I20260927 21:33:56.483815 11899 ts_tablet_manager.cc:1397] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260927 21:33:56.484513 11899 raft_consensus.cc:348] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.484745 11899 raft_consensus.cc:374] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.484834 11899 raft_consensus.cc:729] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02db2c5eb124719b726a49ef2ca6684, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.485059 11899 consensus_queue.cc:260] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.485637 11899 ts_tablet_manager.cc:1428] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:56.485534 11906 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.485775 11906 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.487380 11906 raft_consensus.cc:482] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:56.487499 11906 raft_consensus.cc:504] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.488075 11906 leader_election.cc:288] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307), f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:56.488488 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0a6cf9b1534a69bdab906d1f0c66f5" candidate_uuid: "f02db2c5eb124719b726a49ef2ca6684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d5bcd0e2344759a6398ed4a3590848" is_pre_election: true
I20260927 21:33:56.488517 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0a6cf9b1534a69bdab906d1f0c66f5" candidate_uuid: "f02db2c5eb124719b726a49ef2ca6684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" is_pre_election: true
I20260927 21:33:56.488720 11690 raft_consensus.cc:2454] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f02db2c5eb124719b726a49ef2ca6684 in term 0.
I20260927 21:33:56.488756 11829 raft_consensus.cc:2454] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f02db2c5eb124719b726a49ef2ca6684 in term 0.
I20260927 21:33:56.489115 11477 leader_election.cc:302] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f02db2c5eb124719b726a49ef2ca6684; no voters:
I20260927 21:33:56.489353 11906 raft_consensus.cc:2781] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:56.489480 11906 raft_consensus.cc:482] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:56.489614 11906 raft_consensus.cc:3037] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:56.491324 11906 raft_consensus.cc:504] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.491883 11906 leader_election.cc:288] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [CANDIDATE]: Term 1 election: Requested vote from peers 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307), f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:56.492208 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0a6cf9b1534a69bdab906d1f0c66f5" candidate_uuid: "f02db2c5eb124719b726a49ef2ca6684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d5bcd0e2344759a6398ed4a3590848"
I20260927 21:33:56.492282 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0a6cf9b1534a69bdab906d1f0c66f5" candidate_uuid: "f02db2c5eb124719b726a49ef2ca6684" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
I20260927 21:33:56.492383 11690 raft_consensus.cc:3037] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:56.492434 11829 raft_consensus.cc:3037] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:56.495059 11829 raft_consensus.cc:2454] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f02db2c5eb124719b726a49ef2ca6684 in term 1.
I20260927 21:33:56.495167 11690 raft_consensus.cc:2454] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f02db2c5eb124719b726a49ef2ca6684 in term 1.
I20260927 21:33:56.495484 11479 leader_election.cc:302] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f02db2c5eb124719b726a49ef2ca6684, f45a39838c5b47ea8e77a7b5169e85ee; no voters:
I20260927 21:33:56.495764 11906 raft_consensus.cc:2781] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:56.496085 11906 raft_consensus.cc:686] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [term 1 LEADER]: Becoming Leader. State: Replica: f02db2c5eb124719b726a49ef2ca6684, State: Running, Role: LEADER
I20260927 21:33:56.496462 11906 consensus_queue.cc:237] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.498090 11945 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: TxnStatusTablet state changed. Reason: New leader f02db2c5eb124719b726a49ef2ca6684. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.498353 11945 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:56.499243 11388 catalog_manager.cc:5495] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 reported cstate change: term changed from 0 to 1, leader changed from <none> to f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:56.505120 11947 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:33:56.505330 11947 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:33:56.533454 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.526941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.3.186.193:41143)
0927 21:33:56.527114 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.527119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.527136 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:33:56.527211 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.527215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.527337 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.527471 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:56.527860 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.529023 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.530949 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.531612 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.531746 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.532265 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.532838 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.532955 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.533097 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.533172 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:33:56.533541 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.526588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41143 (local address 127.0.0.1:45470)
0927 21:33:56.526816 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:56.526836 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:56.526957 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:56.527514 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:56.527519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:56.527533 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:56.527712 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.527723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.529183 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.529189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.530803 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.530816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.531780 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.531786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.532000 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.532079 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.532953 (+ 874us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.532978 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:33:56.533077 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:33:56.533216 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:33:56.537307 11829 raft_consensus.cc:1261] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Refusing update from remote peer f02db2c5eb124719b726a49ef2ca6684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:56.537786 11690 raft_consensus.cc:1261] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Refusing update from remote peer f02db2c5eb124719b726a49ef2ca6684: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:56.538036 11945 consensus_queue.cc:1035] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:56.538491 11906 consensus_queue.cc:1035] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:56.549281 11907 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: TxnStatusTablet state changed. Reason: New leader f02db2c5eb124719b726a49ef2ca6684. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.549490 11907 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.551504 11906 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.551760 11906 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:56.555552 11912 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: TxnStatusTablet state changed. Reason: New leader f02db2c5eb124719b726a49ef2ca6684. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.555783 11912 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.555765 11945 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.555994 11945 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f02db2c5eb124719b726a49ef2ca6684: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:33:56.557293 11907 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.557569 11907 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P f45a39838c5b47ea8e77a7b5169e85ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.560319 11912 tablet_replica.cc:439] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f02db2c5eb124719b726a49ef2ca6684" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } } }
I20260927 21:33:56.560525 11912 tablet_replica.cc:442] T 6b0a6cf9b1534a69bdab906d1f0c66f5 P 61d5bcd0e2344759a6398ed4a3590848: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:33:56.566802 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.560301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.3.186.195:33175)
0927 21:33:56.561928 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.561934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.561953 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:33:56.562031 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.562037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.562125 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.562296 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:56.562760 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.563510 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.564805 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.565190 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.565316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.565760 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.566207 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.566349 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.566424 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.566492 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1361,"threads_started":1}
I20260927 21:33:56.581027 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.575471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:50813 (local address 127.3.186.193:41143)
0927 21:33:56.575645 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.575653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.576125 (+ 472us) server_negotiation.cc:407] Connection header received
0927 21:33:56.576241 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.576246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.576318 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.576401 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:56.576774 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.577856 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.579052 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.579822 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.579936 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.580298 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.580421 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.580508 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.580630 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.580703 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:33:56.581037 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.575550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41143 (local address 127.3.186.195:50813)
0927 21:33:56.576027 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:56.576041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:56.576135 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:56.576426 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:56.576429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:56.576438 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:56.576660 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.576666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.577978 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.577982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.578926 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.578936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.579994 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.579998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.580112 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.580154 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.580516 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.580524 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:33:56.580590 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:33:56.580683 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":244,"threads_started":1}
I20260927 21:33:56.589277 11689 raft_consensus.cc:1261] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Refusing update from remote peer f45a39838c5b47ea8e77a7b5169e85ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:56.589320 11551 raft_consensus.cc:1261] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Refusing update from remote peer f45a39838c5b47ea8e77a7b5169e85ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:56.589905 11907 consensus_queue.cc:1035] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:56.590209 11933 consensus_queue.cc:1035] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:56.627770 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.618866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:48335 (local address 127.3.186.195:33175)
0927 21:33:56.619015 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.619021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.622098 (+ 3077us) server_negotiation.cc:407] Connection header received
0927 21:33:56.622298 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.622303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.622381 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.622461 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:56.623019 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.623709 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.625860 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.626546 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.626639 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.627219 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.627306 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.627375 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.627528 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.627597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:33:56.628093 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.618501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:33175 (local address 127.3.186.193:48335)
0927 21:33:56.621996 (+ 3495us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:56.622018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:56.622172 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:56.622491 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:56.622496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:56.622509 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:56.622867 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.622879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.623833 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.623839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.625689 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:56.625706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.626805 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.626812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:56.626976 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.627052 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:56.627408 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:56.627425 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 21:33:56.627503 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:33:56.627633 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3312,"thread_start_us":3178,"threads_started":1}
I20260927 21:33:56.662875 11388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:56.663160 11388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:56.666263 11388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Sending DeleteTablet for 3 replicas of tablet b7fbe916deda42fdb55e557e791bf480
I20260927 21:33:56.666960 11525 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fbe916deda42fdb55e557e791bf480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:56 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45454
I20260927 21:33:56.666949 11670 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fbe916deda42fdb55e557e791bf480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:56 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59904
I20260927 21:33:56.669792 11804 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fbe916deda42fdb55e557e791bf480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:56 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:53874
I20260927 21:33:56.680751 11972 tablet_replica.cc:330] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: stopping tablet replica
I20260927 21:33:56.681013 11972 raft_consensus.cc:2229] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:56.681277 11972 raft_consensus.cc:2258] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:56.682294 11972 ts_tablet_manager.cc:1905] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:56.689495 11972 ts_tablet_manager.cc:1918] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:56.689719 11972 log.cc:1196] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/b7fbe916deda42fdb55e557e791bf480
I20260927 21:33:56.690235 11972 ts_tablet_manager.cc:1939] T b7fbe916deda42fdb55e557e791bf480 P 61d5bcd0e2344759a6398ed4a3590848: Deleting consensus metadata
I20260927 21:33:56.691284 11374 catalog_manager.cc:4841] TS 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307): tablet b7fbe916deda42fdb55e557e791bf480 (table test-table [id=20daf49f398e4bf4be67b0c952773925]) successfully deleted
I20260927 21:33:56.692109 11974 tablet_replica.cc:330] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: stopping tablet replica
I20260927 21:33:56.692305 11974 raft_consensus.cc:2229] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:56.692859 11974 raft_consensus.cc:2258] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:56.693774 11974 ts_tablet_manager.cc:1905] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:56.700796 11973 tablet_replica.cc:330] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: stopping tablet replica
I20260927 21:33:56.701028 11973 raft_consensus.cc:2229] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:56.701395 11973 raft_consensus.cc:2258] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:56.702476 11973 ts_tablet_manager.cc:1905] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:56.712313 11974 ts_tablet_manager.cc:1918] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:56.714818 11973 ts_tablet_manager.cc:1918] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:56.715000 11974 log.cc:1196] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/b7fbe916deda42fdb55e557e791bf480
I20260927 21:33:56.715562 11974 ts_tablet_manager.cc:1939] T b7fbe916deda42fdb55e557e791bf480 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting consensus metadata
I20260927 21:33:56.716682 11376 catalog_manager.cc:4841] TS f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175): tablet b7fbe916deda42fdb55e557e791bf480 (table test-table [id=20daf49f398e4bf4be67b0c952773925]) successfully deleted
I20260927 21:33:56.717049 11973 log.cc:1196] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/b7fbe916deda42fdb55e557e791bf480
I20260927 21:33:56.717648 11973 ts_tablet_manager.cc:1939] T b7fbe916deda42fdb55e557e791bf480 P f02db2c5eb124719b726a49ef2ca6684: Deleting consensus metadata
I20260927 21:33:56.718673 11376 catalog_manager.cc:4841] TS f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143): tablet b7fbe916deda42fdb55e557e791bf480 (table test-table [id=20daf49f398e4bf4be67b0c952773925]) successfully deleted
Sep 27 21:33:56 dist-test-slave-pqr8 krb5kdc[11344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:33:56 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:33:56 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:33:56.748791 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:33:56.735776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.3.186.254:33897)
0927 21:33:56.735982 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.735991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.736342 (+ 351us) server_negotiation.cc:407] Connection header received
0927 21:33:56.736620 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.736626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.736706 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.736809 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:56.738424 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.739532 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.741407 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.742084 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.742210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.748552 (+ 6342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34182: BlockingRecv error: recv got EOF from 127.0.0.1:34182 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 21:33:56 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM
I20260927 21:33:56.774957 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:56.755161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.3.186.254:33897)
0927 21:33:56.755698 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:56.755704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:56.757983 (+ 2279us) server_negotiation.cc:407] Connection header received
0927 21:33:56.758135 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:56.758141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:56.758209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:56.758307 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:56.759860 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.760952 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.762751 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:56.763426 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:56.763551 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:56.768121 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:56.768128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:56.768133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:56.768170 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:56.771539 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:56.772293 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:56.772298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:56.772302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:56.772443 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:56.772847 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:56.772852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:56.772855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:56.773171 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:56.773416 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:56.774389 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 21:33:56.774622 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260927 21:33:56.778400 11388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:56.779027 11388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:56.785964 11525 tablet_service.cc:1467] Processing CreateTablet for tablet 6b70d27198d640ce98375955205560f8 (DEFAULT_TABLE table=test-table [id=c1533282579d4868ba889de042438fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:56.786419 11525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b70d27198d640ce98375955205560f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.786355 11804 tablet_service.cc:1467] Processing CreateTablet for tablet 6b70d27198d640ce98375955205560f8 (DEFAULT_TABLE table=test-table [id=c1533282579d4868ba889de042438fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:56.786734 11804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b70d27198d640ce98375955205560f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.790441 11670 tablet_service.cc:1467] Processing CreateTablet for tablet 6b70d27198d640ce98375955205560f8 (DEFAULT_TABLE table=test-table [id=c1533282579d4868ba889de042438fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:56.790818 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b70d27198d640ce98375955205560f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:56.793171 11899 tablet_bootstrap.cc:492] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Bootstrap starting.
I20260927 21:33:56.794817 11899 tablet_bootstrap.cc:654] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.802807 11901 tablet_bootstrap.cc:492] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Bootstrap starting.
I20260927 21:33:56.804342 11901 tablet_bootstrap.cc:654] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.805292 11899 tablet_bootstrap.cc:492] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: No bootstrap required, opened a new log
I20260927 21:33:56.805420 11899 ts_tablet_manager.cc:1397] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260927 21:33:56.806186 11899 raft_consensus.cc:348] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.806529 11899 raft_consensus.cc:374] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.806679 11899 raft_consensus.cc:729] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02db2c5eb124719b726a49ef2ca6684, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.806990 11899 consensus_queue.cc:260] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.807633 11899 ts_tablet_manager.cc:1428] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:56.810873 11900 tablet_bootstrap.cc:492] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Bootstrap starting.
I20260927 21:33:56.812312 11900 tablet_bootstrap.cc:654] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:56.816474 11901 tablet_bootstrap.cc:492] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: No bootstrap required, opened a new log
I20260927 21:33:56.816612 11901 ts_tablet_manager.cc:1397] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20260927 21:33:56.817343 11901 raft_consensus.cc:348] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.817612 11901 raft_consensus.cc:374] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.817710 11901 raft_consensus.cc:729] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.817915 11901 consensus_queue.cc:260] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.818413 11901 ts_tablet_manager.cc:1428] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:33:56.827152 11900 tablet_bootstrap.cc:492] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: No bootstrap required, opened a new log
I20260927 21:33:56.827359 11900 ts_tablet_manager.cc:1397] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20260927 21:33:56.828238 11900 raft_consensus.cc:348] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.828620 11900 raft_consensus.cc:374] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:56.828763 11900 raft_consensus.cc:729] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45a39838c5b47ea8e77a7b5169e85ee, State: Initialized, Role: FOLLOWER
I20260927 21:33:56.829079 11900 consensus_queue.cc:260] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:56.829723 11900 ts_tablet_manager.cc:1428] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 21:33:57.233214 11912 raft_consensus.cc:482] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:57.233371 11912 raft_consensus.cc:504] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.233969 11912 leader_election.cc:288] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143), f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:57.234512 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b70d27198d640ce98375955205560f8" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" is_pre_election: true
I20260927 21:33:57.234493 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b70d27198d640ce98375955205560f8" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684" is_pre_election: true
I20260927 21:33:57.234732 11829 raft_consensus.cc:2454] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 0.
I20260927 21:33:57.234733 11551 raft_consensus.cc:2454] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 0.
I20260927 21:33:57.235176 11619 leader_election.cc:302] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f45a39838c5b47ea8e77a7b5169e85ee; no voters:
I20260927 21:33:57.235446 11912 raft_consensus.cc:2781] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:57.235572 11912 raft_consensus.cc:482] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:57.235664 11912 raft_consensus.cc:3037] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:57.237872 11912 raft_consensus.cc:504] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.238354 11912 leader_election.cc:288] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 election: Requested vote from peers f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143), f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:57.238824 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b70d27198d640ce98375955205560f8" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684"
I20260927 21:33:57.238883 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b70d27198d640ce98375955205560f8" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
I20260927 21:33:57.239039 11551 raft_consensus.cc:3037] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:57.239053 11829 raft_consensus.cc:3037] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:57.241802 11551 raft_consensus.cc:2454] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 1.
I20260927 21:33:57.241802 11829 raft_consensus.cc:2454] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 1.
I20260927 21:33:57.242153 11619 leader_election.cc:302] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f02db2c5eb124719b726a49ef2ca6684; no voters:
I20260927 21:33:57.242414 11912 raft_consensus.cc:2781] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:57.242794 11912 raft_consensus.cc:686] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 LEADER]: Becoming Leader. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Running, Role: LEADER
I20260927 21:33:57.243075 11912 consensus_queue.cc:237] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.245580 11387 catalog_manager.cc:5495] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "61d5bcd0e2344759a6398ed4a3590848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:57.261632 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.256940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.3.186.194:41307)
0927 21:33:57.257383 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:57.257388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:57.257400 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:33:57.257442 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:57.257445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:57.257497 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:57.257641 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:57.257993 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.258707 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.260003 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.260394 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.260520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.260906 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:57.261009 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:57.261100 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:57.261234 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 21:33:57.261305 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":224,"threads_started":1}
I20260927 21:33:57.264021 11829 raft_consensus.cc:1261] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Refusing update from remote peer 61d5bcd0e2344759a6398ed4a3590848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:57.264107 11551 raft_consensus.cc:1261] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Refusing update from remote peer 61d5bcd0e2344759a6398ed4a3590848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:57.264559 11990 consensus_queue.cc:1035] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:57.264832 11912 consensus_queue.cc:1035] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:57.279381 11388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:57.279560 11388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:57.281980 11388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Sending DeleteTablet for 3 replicas of tablet 6b70d27198d640ce98375955205560f8
I20260927 21:33:57.282639 11525 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b70d27198d640ce98375955205560f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:57 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45454
I20260927 21:33:57.282819 11670 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b70d27198d640ce98375955205560f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:57 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59904
I20260927 21:33:57.282873 11804 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b70d27198d640ce98375955205560f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:57 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:53874
I20260927 21:33:57.283981 11388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:33:57.284791 11388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:33:57.286360 12003 tablet_replica.cc:330] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: stopping tablet replica
I20260927 21:33:57.286525 12003 raft_consensus.cc:2229] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:57.286765 12003 raft_consensus.cc:2258] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:57.287774 12004 tablet_replica.cc:330] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: stopping tablet replica
I20260927 21:33:57.287979 12004 raft_consensus.cc:2229] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:57.288074 12003 ts_tablet_manager.cc:1905] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:57.288331 12004 raft_consensus.cc:2258] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:57.288916 12005 tablet_replica.cc:330] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: stopping tablet replica
I20260927 21:33:57.289078 12005 raft_consensus.cc:2229] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:57.289393 12005 raft_consensus.cc:2258] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:57.290423 12005 ts_tablet_manager.cc:1905] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:57.292410 11525 tablet_service.cc:1467] Processing CreateTablet for tablet ae7dcbaca1da4b3ba7cd2951d8874235 (DEFAULT_TABLE table=test-table [id=c71f24672ec745d685e7a52534c42e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:57.292855 11525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7dcbaca1da4b3ba7cd2951d8874235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:57.293129 11804 tablet_service.cc:1467] Processing CreateTablet for tablet ae7dcbaca1da4b3ba7cd2951d8874235 (DEFAULT_TABLE table=test-table [id=c71f24672ec745d685e7a52534c42e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:57.293495 11804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7dcbaca1da4b3ba7cd2951d8874235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:57.294667 12003 ts_tablet_manager.cc:1918] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:57.294795 12003 log.cc:1196] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/6b70d27198d640ce98375955205560f8
I20260927 21:33:57.295217 12003 ts_tablet_manager.cc:1939] T 6b70d27198d640ce98375955205560f8 P f02db2c5eb124719b726a49ef2ca6684: Deleting consensus metadata
I20260927 21:33:57.296381 12005 ts_tablet_manager.cc:1918] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:57.296517 12005 log.cc:1196] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/6b70d27198d640ce98375955205560f8
I20260927 21:33:57.296939 12005 ts_tablet_manager.cc:1939] T 6b70d27198d640ce98375955205560f8 P 61d5bcd0e2344759a6398ed4a3590848: Deleting consensus metadata
I20260927 21:33:57.297155 11376 catalog_manager.cc:4841] TS f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143): tablet 6b70d27198d640ce98375955205560f8 (table test-table [id=c1533282579d4868ba889de042438fac]) successfully deleted
I20260927 21:33:57.297829 11374 catalog_manager.cc:4841] TS 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307): tablet 6b70d27198d640ce98375955205560f8 (table test-table [id=c1533282579d4868ba889de042438fac]) successfully deleted
I20260927 21:33:57.299418 11900 tablet_bootstrap.cc:492] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Bootstrap starting.
I20260927 21:33:57.300012 11899 tablet_bootstrap.cc:492] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Bootstrap starting.
I20260927 21:33:57.300101 12004 ts_tablet_manager.cc:1905] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:57.300765 11900 tablet_bootstrap.cc:654] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:57.301631 11899 tablet_bootstrap.cc:654] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:57.301790 11670 tablet_service.cc:1467] Processing CreateTablet for tablet ae7dcbaca1da4b3ba7cd2951d8874235 (DEFAULT_TABLE table=test-table [id=c71f24672ec745d685e7a52534c42e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:33:57.302153 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7dcbaca1da4b3ba7cd2951d8874235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:33:57.302544 11900 tablet_bootstrap.cc:492] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: No bootstrap required, opened a new log
I20260927 21:33:57.302702 11900 ts_tablet_manager.cc:1397] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:33:57.303308 11900 raft_consensus.cc:348] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.303498 11900 raft_consensus.cc:374] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:57.303544 11899 tablet_bootstrap.cc:492] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: No bootstrap required, opened a new log
I20260927 21:33:57.303689 11899 ts_tablet_manager.cc:1397] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:33:57.303593 11900 raft_consensus.cc:729] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45a39838c5b47ea8e77a7b5169e85ee, State: Initialized, Role: FOLLOWER
I20260927 21:33:57.304042 11900 consensus_queue.cc:260] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.304306 11899 raft_consensus.cc:348] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.304504 11900 ts_tablet_manager.cc:1428] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 21:33:57.304515 11899 raft_consensus.cc:374] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:57.304582 11899 raft_consensus.cc:729] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02db2c5eb124719b726a49ef2ca6684, State: Initialized, Role: FOLLOWER
I20260927 21:33:57.304805 11899 consensus_queue.cc:260] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.305243 11899 ts_tablet_manager.cc:1428] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:33:57.308935 12004 ts_tablet_manager.cc:1918] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:33:57.309060 12004 log.cc:1196] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/6b70d27198d640ce98375955205560f8
I20260927 21:33:57.309443 12004 ts_tablet_manager.cc:1939] T 6b70d27198d640ce98375955205560f8 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting consensus metadata
I20260927 21:33:57.310339 11376 catalog_manager.cc:4841] TS f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175): tablet 6b70d27198d640ce98375955205560f8 (table test-table [id=c1533282579d4868ba889de042438fac]) successfully deleted
I20260927 21:33:57.313421 11901 tablet_bootstrap.cc:492] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Bootstrap starting.
I20260927 21:33:57.314548 11901 tablet_bootstrap.cc:654] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Neither blocks nor log segments found. Creating new log.
I20260927 21:33:57.316309 11901 tablet_bootstrap.cc:492] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: No bootstrap required, opened a new log
I20260927 21:33:57.316424 11901 ts_tablet_manager.cc:1397] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:33:57.317063 11901 raft_consensus.cc:348] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.317235 11901 raft_consensus.cc:374] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:33:57.317286 11901 raft_consensus.cc:729] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Initialized, Role: FOLLOWER
I20260927 21:33:57.317479 11901 consensus_queue.cc:260] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.317934 11901 ts_tablet_manager.cc:1428] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 21:33:57.570446 12002 raft_consensus.cc:482] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:33:57.570629 12002 raft_consensus.cc:504] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.571321 12002 leader_election.cc:288] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143), f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:57.571759 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7dcbaca1da4b3ba7cd2951d8874235" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684" is_pre_election: true
I20260927 21:33:57.571830 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7dcbaca1da4b3ba7cd2951d8874235" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" is_pre_election: true
I20260927 21:33:57.572007 11551 raft_consensus.cc:2454] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 0.
I20260927 21:33:57.572055 11829 raft_consensus.cc:2454] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 0.
I20260927 21:33:57.572438 11619 leader_election.cc:302] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f02db2c5eb124719b726a49ef2ca6684; no voters:
I20260927 21:33:57.572700 12002 raft_consensus.cc:2781] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:33:57.572793 12002 raft_consensus.cc:482] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:33:57.572913 12002 raft_consensus.cc:3037] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:57.575094 12002 raft_consensus.cc:504] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.575588 12002 leader_election.cc:288] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 election: Requested vote from peers f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143), f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175)
I20260927 21:33:57.576076 11551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7dcbaca1da4b3ba7cd2951d8874235" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02db2c5eb124719b726a49ef2ca6684"
I20260927 21:33:57.576161 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7dcbaca1da4b3ba7cd2951d8874235" candidate_uuid: "61d5bcd0e2344759a6398ed4a3590848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45a39838c5b47ea8e77a7b5169e85ee"
I20260927 21:33:57.576269 11551 raft_consensus.cc:3037] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:57.576355 11829 raft_consensus.cc:3037] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:33:57.579057 11551 raft_consensus.cc:2454] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 1.
I20260927 21:33:57.579162 11829 raft_consensus.cc:2454] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d5bcd0e2344759a6398ed4a3590848 in term 1.
I20260927 21:33:57.579483 11619 leader_election.cc:302] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d5bcd0e2344759a6398ed4a3590848, f02db2c5eb124719b726a49ef2ca6684; no voters:
I20260927 21:33:57.579761 12002 raft_consensus.cc:2781] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:33:57.579914 12002 raft_consensus.cc:686] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 LEADER]: Becoming Leader. State: Replica: 61d5bcd0e2344759a6398ed4a3590848, State: Running, Role: LEADER
I20260927 21:33:57.580183 12002 consensus_queue.cc:237] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } }
I20260927 21:33:57.582448 11388 catalog_manager.cc:5495] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194). New cstate: current_term: 1 leader_uuid: "61d5bcd0e2344759a6398ed4a3590848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d5bcd0e2344759a6398ed4a3590848" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 41307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 } health_report { overall_health: UNKNOWN } } }
I20260927 21:33:57.611430 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.606404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:40171 (local address 127.3.186.193:41143)
0927 21:33:57.606899 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:57.606904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:57.607177 (+ 273us) server_negotiation.cc:407] Connection header received
0927 21:33:57.607348 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:57.607351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:57.607413 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:57.607500 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:57.607948 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.608646 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.610025 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.610432 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.610523 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.610854 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:57.610936 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:57.611033 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:57.611144 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:33:57.611200 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":228,"threads_started":1}
I20260927 21:33:57.611464 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.606465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41143 (local address 127.3.186.194:40171)
0927 21:33:57.607049 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:57.607079 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:57.607210 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:57.607539 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:57.607542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:57.607551 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:57.607806 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:57.607814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.608787 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.608791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:57.609912 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:57.609922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.610550 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.610554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:57.610662 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.610712 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:57.611038 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:57.611045 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:33:57.611106 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:33:57.611193 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":328,"threads_started":1}
I20260927 21:33:57.617906 11829 raft_consensus.cc:1261] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Refusing update from remote peer 61d5bcd0e2344759a6398ed4a3590848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:57.617965 11551 raft_consensus.cc:1261] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Refusing update from remote peer 61d5bcd0e2344759a6398ed4a3590848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:33:57.618358 11990 consensus_queue.cc:1035] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45a39838c5b47ea8e77a7b5169e85ee" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:57.618582 12002 consensus_queue.cc:1035] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02db2c5eb124719b726a49ef2ca6684" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:33:57.641973 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.636584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:55105 (local address 127.3.186.194:41307)
0927 21:33:57.636730 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:57.636734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:57.637437 (+ 703us) server_negotiation.cc:407] Connection header received
0927 21:33:57.637617 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:57.637622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:57.637697 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:57.637790 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:33:57.638191 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.638838 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.640466 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.641040 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.641130 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.641436 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:33:57.641560 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:33:57.641645 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:57.641750 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:33:57.641803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:33:57.642165 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.636478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:41307 (local address 127.3.186.193:55105)
0927 21:33:57.637332 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 21:33:57.637349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:33:57.637473 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:33:57.637839 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:33:57.637843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:33:57.637862 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:33:57.638060 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:57.638067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.638934 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.638937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:57.640302 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:33:57.640314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.641175 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.641179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:33:57.641288 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.641318 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:33:57.641658 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:33:57.641666 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:33:57.641720 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:33:57.641805 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":559,"threads_started":1}
I20260927 21:33:57.662072 11389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:57.662274 11389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:33:57.664611 11389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83bb2f411041eb9343ae4a601c6688: Sending DeleteTablet for 3 replicas of tablet ae7dcbaca1da4b3ba7cd2951d8874235
I20260927 21:33:57.665262 11525 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7dcbaca1da4b3ba7cd2951d8874235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:57 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45454
I20260927 21:33:57.665337 11670 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7dcbaca1da4b3ba7cd2951d8874235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:57 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59904
I20260927 21:33:57.665606 12005 tablet_replica.cc:330] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: stopping tablet replica
I20260927 21:33:57.665633 11804 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7dcbaca1da4b3ba7cd2951d8874235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:33:57 UTC) from {username='oryx', principal='oryx/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:53874
I20260927 21:33:57.665813 12005 raft_consensus.cc:2229] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:33:57.665867 12003 tablet_replica.cc:330] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: stopping tablet replica
I20260927 21:33:57.666021 12003 raft_consensus.cc:2229] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:57.666080 12005 raft_consensus.cc:2258] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:57.666181 12003 raft_consensus.cc:2258] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:57.666222 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 11460
I20260927 21:33:57.666416 12004 tablet_replica.cc:330] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: stopping tablet replica
I20260927 21:33:57.666607 12004 raft_consensus.cc:2229] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:33:57.666769 12004 raft_consensus.cc:2258] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:33:57.667325 12005 ts_tablet_manager.cc:1905] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:57.668226 12003 ts_tablet_manager.cc:1905] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:57.673615 12005 ts_tablet_manager.cc:1918] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:57.673766 12005 log.cc:1196] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/ae7dcbaca1da4b3ba7cd2951d8874235
I20260927 21:33:57.674144 12004 ts_tablet_manager.cc:1905] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:33:57.674221 12005 ts_tablet_manager.cc:1939] T ae7dcbaca1da4b3ba7cd2951d8874235 P 61d5bcd0e2344759a6398ed4a3590848: Deleting consensus metadata
I20260927 21:33:57.675141 11374 catalog_manager.cc:4841] TS 61d5bcd0e2344759a6398ed4a3590848 (127.3.186.194:41307): tablet ae7dcbaca1da4b3ba7cd2951d8874235 (table test-table [id=c71f24672ec745d685e7a52534c42e43]) successfully deleted
Sep 27 21:33:57 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.193@KRBTEST.COM
I20260927 21:33:57.678923 12004 ts_tablet_manager.cc:1918] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:57.679029 12004 log.cc:1196] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/ae7dcbaca1da4b3ba7cd2951d8874235
I20260927 21:33:57.679435 12004 ts_tablet_manager.cc:1939] T ae7dcbaca1da4b3ba7cd2951d8874235 P f45a39838c5b47ea8e77a7b5169e85ee: Deleting consensus metadata
I20260927 21:33:57.680241 11376 catalog_manager.cc:4841] TS f45a39838c5b47ea8e77a7b5169e85ee (127.3.186.195:33175): tablet ae7dcbaca1da4b3ba7cd2951d8874235 (table test-table [id=c71f24672ec745d685e7a52534c42e43]) successfully deleted
I20260927 21:33:57.680987 12003 ts_tablet_manager.cc:1918] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:33:57.681140 12003 log.cc:1196] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/ae7dcbaca1da4b3ba7cd2951d8874235
I20260927 21:33:57.681478 12003 ts_tablet_manager.cc:1939] T ae7dcbaca1da4b3ba7cd2951d8874235 P f02db2c5eb124719b726a49ef2ca6684: Deleting consensus metadata
I20260927 21:33:57.682016 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.667344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.3.186.193:41143)
0927 21:33:57.667527 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:57.667532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:57.667728 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:33:57.667920 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:57.667925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:57.667994 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:57.668094 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:57.669230 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.670402 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.671670 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.672091 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.672246 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.676654 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:57.676669 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:57.676694 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:57.676721 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:57.679504 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:57.680130 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:57.680139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:57.680143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:57.680265 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:57.680633 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:57.680640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:57.680643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:57.680989 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:57.681285 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:57.681489 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 21:33:57.681734 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:33:57.682448 11376 catalog_manager.cc:4841] TS f02db2c5eb124719b726a49ef2ca6684 (127.3.186.193:41143): tablet ae7dcbaca1da4b3ba7cd2951d8874235 (table test-table [id=c71f24672ec745d685e7a52534c42e43]) successfully deleted
I20260927 21:33:57.682668 11591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:57.804314 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 11460
I20260927 21:33:57.826241 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 11600
Sep 27 21:33:57 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.194@KRBTEST.COM
I20260927 21:33:57.839932 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.826914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.3.186.194:41307)
0927 21:33:57.827128 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:57.827137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:57.827192 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:33:57.827411 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:57.827415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:57.827486 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:57.827590 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:57.828763 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.829436 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.830704 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.831108 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.831232 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.834930 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:57.834942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:57.834956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:57.834988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:57.837684 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:57.838258 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:57.838264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:57.838266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:57.838379 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:57.838661 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:57.838664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:57.838666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:57.838991 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:57.839198 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:57.839412 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:33:57.839558 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:33:57.840490 11730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:57.961202 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 11600
I20260927 21:33:57.982534 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 11739
Sep 27 21:33:57 dist-test-slave-pqr8 krb5kdc[11344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.186.195@KRBTEST.COM
I20260927 21:33:58.003237 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:33:57.983212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.3.186.195:33175)
0927 21:33:57.983597 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 21:33:57.983601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:33:57.983623 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:33:57.983673 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:33:57.983677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:33:57.983737 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:33:57.983817 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:33:57.985153 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.985872 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.987012 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:33:57.987395 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:33:57.987480 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:33:57.998438 (+ 10958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:33:57.998450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:33:57.998466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:33:57.998492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:33:58.000976 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:58.001500 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:58.001506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:58.001547 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:58.001682 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:33:58.001962 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:33:58.001965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:33:58.001967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:33:58.002268 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:33:58.002472 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 21:33:58.002688 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:33:58.002866 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":214,"threads_started":1}
I20260927 21:33:58.003921 11869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:58.120920 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 11739
I20260927 21:33:58.141872 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 11357
I20260927 21:33:58.142899 11429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:33:58.249953 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 11357
2026-09-27T21:33:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6984 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:58 dist-test-slave-pqr8 krb5kdc[12029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:58 dist-test-slave-pqr8 krb5kdc[12029](info): set up 2 sockets
Sep 27 21:33:58 dist-test-slave-pqr8 krb5kdc[12029](info): commencing operation
krb5kdc: starting...
W20260927 21:33:59.302387 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:33:59 dist-test-slave-pqr8 krb5kdc[12029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:33:59.486897 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:33483
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:33483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:33:59.781060 12042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:33:59.781378 12042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:33:59.781566 12042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:33:59.791910 12042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:33:59.792037 12042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:33:59.792107 12042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:33:59.792186 12042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:33:59.792250 12042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:33:59.801270 12042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:33483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:33483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:33:59.802439 12042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:33:59.803609 12042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:33:59.814421 12047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:59.814392 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:33:59.814469 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:33:59.815146 12042 server_base.cc:1034] running on GCE node
I20260927 21:33:59.815822 12042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:33:59.817011 12042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:33:59.818225 12042 hybrid_clock.cc:648] HybridClock initialized: now 1790544839818161 us; error 51 us; skew 500 ppm
I20260927 21:33:59.818549 12042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T21:33:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1627 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:33:59 dist-test-slave-pqr8 krb5kdc[12056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:33:59 dist-test-slave-pqr8 krb5kdc[12056](info): set up 2 sockets
Sep 27 21:33:59 dist-test-slave-pqr8 krb5kdc[12056](info): commencing operation
krb5kdc: starting...
W20260927 21:34:00.922423 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:34:01 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:34:01.104269 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44171
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:01.398722 12069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:01.398932 12069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:01.399061 12069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:01.405692 12069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:34:01.405802 12069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:01.405927 12069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:01.406042 12069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:34:01.406118 12069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:34:01.415162 12069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:01.416217 12069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:01.417387 12069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:01.428253 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:01.428193 12075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:01.428294 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:01.429033 12069 server_base.cc:1034] running on GCE node
I20260927 21:34:01.429759 12069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:01.430958 12069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:01.432144 12069 hybrid_clock.cc:648] HybridClock initialized: now 1790544841432105 us; error 36 us; skew 500 ppm
I20260927 21:34:01.432482 12069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:01 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:01.435607 12069 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:34:01.436959 12069 webserver.cc:460] Webserver started at http://127.3.186.254:43983/ using document root <none> and password file <none>
I20260927 21:34:01.437628 12069 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:01.437753 12069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:01.438063 12069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:01.440716 12069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "176be78961e845a08e466b99afcda311"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "ad168bb7584577f1f9ae61806cb2f745"
server_key_iv: "0835312d8afc421cfc7c02cab4c0c2a3"
server_key_version: "encryptionkey@0"
I20260927 21:34:01.441455 12069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "176be78961e845a08e466b99afcda311"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "ad168bb7584577f1f9ae61806cb2f745"
server_key_iv: "0835312d8afc421cfc7c02cab4c0c2a3"
server_key_version: "encryptionkey@0"
I20260927 21:34:01.445885 12069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:34:01.448941 12084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:01.450222 12069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:34:01.450413 12069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "176be78961e845a08e466b99afcda311"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "ad168bb7584577f1f9ae61806cb2f745"
server_key_iv: "0835312d8afc421cfc7c02cab4c0c2a3"
server_key_version: "encryptionkey@0"
I20260927 21:34:01.450564 12069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:01.463699 12069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:01.467484 12069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:01.467749 12069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:01.484869 12069 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44171
I20260927 21:34:01.484884 12135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44171 every 8 connection(s)
I20260927 21:34:01.485978 12069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:34:01.488880 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12069
I20260927 21:34:01.489063 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:34:01.489409 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:01.489835 3819 external_mini_cluster.cc:1428] Setting key 873ca19d726f5ddbd3844baa4698dd6f
I20260927 21:34:01.503362 12136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Bootstrap starting.
Sep 27 21:34:01 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:01.506743 12136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:01.507973 12136 log.cc:824] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:01.510957 12136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: No bootstrap required, opened a new log
I20260927 21:34:01.512723 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:01.491703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.3.186.254:44171)
0927 21:34:01.492489 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:01.492498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:01.492536 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:34:01.494124 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:01.494142 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:01.494469 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:01.494754 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:01.496400 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:01.498103 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:01.499749 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:01.500388 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:01.500515 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:01.504930 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:01.504942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:01.504960 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:01.505004 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:01.508633 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:01.509397 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:01.509403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:01.509409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:01.509597 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:01.510006 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:01.510011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:01.510014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:01.510617 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:01.510876 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:01.511205 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 21:34:01.511579 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":389,"threads_started":1}
I20260927 21:34:01.518512 12136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:01.518862 12136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:01.518971 12136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176be78961e845a08e466b99afcda311, State: Initialized, Role: FOLLOWER
I20260927 21:34:01.519507 12136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:01.519728 12136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:34:01.519848 12136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:34:01.520028 12136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:01.521963 12136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:01.522392 12136 leader_election.cc:302] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 176be78961e845a08e466b99afcda311; no voters:
I20260927 21:34:01.522856 12136 leader_election.cc:288] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:34:01.523101 12141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:34:01.523699 12141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 LEADER]: Becoming Leader. State: Replica: 176be78961e845a08e466b99afcda311, State: Running, Role: LEADER
I20260927 21:34:01.524133 12141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:01.524595 12136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:34:01.526661 12142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "176be78961e845a08e466b99afcda311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } } }
I20260927 21:34:01.526901 12143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176be78961e845a08e466b99afcda311. Latest consensus state: current_term: 1 leader_uuid: "176be78961e845a08e466b99afcda311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } } }
I20260927 21:34:01.527103 12142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:01.527161 12143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:01.530748 12147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:34:01.535468 12147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:34:01.542856 12147 catalog_manager.cc:1347] Generated new cluster ID: c6a8aceca5d14801a2344561f2d99aee
I20260927 21:34:01.542946 12147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:34:01.554483 12147 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:34:01.556124 12147 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:34:01.567641 12147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Generated new TSK 0
I20260927 21:34:01.568424 12147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:34:01.619264 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44171
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:01.920325 12162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:01.920583 12162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:01.920738 12162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:01.927495 12162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:01.927613 12162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:01.927779 12162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:34:01.936285 12162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:01.937345 12162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:01.938480 12162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:01.950093 12168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:01.950148 12170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:01.950356 12167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:01.951121 12162 server_base.cc:1034] running on GCE node
I20260927 21:34:01.951623 12162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:01.952423 12162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:01.953598 12162 hybrid_clock.cc:648] HybridClock initialized: now 1790544841953558 us; error 47 us; skew 500 ppm
I20260927 21:34:01.953877 12162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:01 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:01.956740 12162 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:34:01.958003 12162 webserver.cc:460] Webserver started at http://127.3.186.193:36355/ using document root <none> and password file <none>
I20260927 21:34:01.958573 12162 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:01.958704 12162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:01.958973 12162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:01.961493 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "4cbf4434ae2e40fb8ea18a84afc862ff"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "01a1f4301427b79015dd9bf4eecf28bb"
server_key_iv: "a09f0441b83af146932d44d60631af4b"
server_key_version: "encryptionkey@0"
I20260927 21:34:01.962275 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "4cbf4434ae2e40fb8ea18a84afc862ff"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "01a1f4301427b79015dd9bf4eecf28bb"
server_key_iv: "a09f0441b83af146932d44d60631af4b"
server_key_version: "encryptionkey@0"
I20260927 21:34:01.966670 12162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 21:34:01.969774 12177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:01.971117 12162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:34:01.971292 12162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "4cbf4434ae2e40fb8ea18a84afc862ff"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "01a1f4301427b79015dd9bf4eecf28bb"
server_key_iv: "a09f0441b83af146932d44d60631af4b"
server_key_version: "encryptionkey@0"
I20260927 21:34:01.971493 12162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:01.990556 12162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:01.993999 12162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:01.994246 12162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:01.995070 12162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:01.996224 12162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:01.996335 12162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:01.996466 12162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:01.996534 12162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.027114 12162 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:39319
I20260927 21:34:02.027185 12289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:39319 every 8 connection(s)
I20260927 21:34:02.028229 12162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:34:02.036396 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12162
I20260927 21:34:02.036571 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:34:02.037168 3819 external_mini_cluster.cc:1428] Setting key 2b8bde1a3e0d9dba3ff7b1dec4e50291
Sep 27 21:34:02 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:02.047214 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:02.030485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:39359 (local address 127.3.186.254:44171)
0927 21:34:02.030899 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:02.030904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:02.031789 (+ 885us) server_negotiation.cc:407] Connection header received
0927 21:34:02.033537 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:02.033543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:02.033621 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:02.033745 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:02.035017 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.035686 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.037259 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.037735 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.037880 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:02.041937 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:02.041942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:02.041945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:02.041973 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:02.044358 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:02.045058 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:02.045065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:02.045068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:02.045183 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:02.045632 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:02.045637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:02.045641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:02.045901 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:02.046069 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:02.046658 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 21:34:02.046863 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":187,"threads_started":1}
I20260927 21:34:02.048076 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:02.030981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:39359)
0927 21:34:02.031641 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:02.031683 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:02.033272 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:02.033959 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:02.033967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:02.034398 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:02.034857 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:02.034872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.035831 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.035837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:02.037120 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:02.037132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.037969 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.037975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:02.038185 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:02.039150 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:02.039206 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:02.041725 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:02.044488 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:02.044496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:02.044509 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:02.044941 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:02.045289 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:02.045295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:02.045299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:02.045474 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:02.046032 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:02.046050 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:02.046347 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 21:34:02.046589 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 21:34:02.047014 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260927 21:34:02.049474 12290 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44171
I20260927 21:34:02.049777 12290 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:02.050416 12290 heartbeater.cc:511] Master 127.3.186.254:44171 requested a full tablet report, sending...
I20260927 21:34:02.052522 12101 ts_manager.cc:194] Registered new tserver with Master: 4cbf4434ae2e40fb8ea18a84afc862ff (127.3.186.193:39319)
I20260927 21:34:02.054718 12101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:39359
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:34:02.082068 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44171
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:02.380736 12296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:02.380966 12296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:02.381129 12296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:02.387975 12296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:02.388068 12296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:02.388245 12296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:34:02.396994 12296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:02.398162 12296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:02.399328 12296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:02.411312 12304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:02.411238 12302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:02.411353 12301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:02.411921 12296 server_base.cc:1034] running on GCE node
I20260927 21:34:02.412498 12296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:02.413293 12296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:02.414512 12296 hybrid_clock.cc:648] HybridClock initialized: now 1790544842414475 us; error 33 us; skew 500 ppm
I20260927 21:34:02.414799 12296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:02 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:02.417591 12296 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:34:02.418781 12296 webserver.cc:460] Webserver started at http://127.3.186.194:45115/ using document root <none> and password file <none>
I20260927 21:34:02.419343 12296 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:02.419461 12296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:02.419715 12296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:02.422322 12296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "d6a17b8bf2fa4bc19b9ef4be53709ab3"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "86a3e4d34c0a1c9bad73449f500c2663"
server_key_iv: "559586694ef12b56749052c176c46541"
server_key_version: "encryptionkey@0"
I20260927 21:34:02.422983 12296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "d6a17b8bf2fa4bc19b9ef4be53709ab3"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "86a3e4d34c0a1c9bad73449f500c2663"
server_key_iv: "559586694ef12b56749052c176c46541"
server_key_version: "encryptionkey@0"
I20260927 21:34:02.427685 12296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 21:34:02.430701 12311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.432087 12296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:34:02.432240 12296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "d6a17b8bf2fa4bc19b9ef4be53709ab3"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "86a3e4d34c0a1c9bad73449f500c2663"
server_key_iv: "559586694ef12b56749052c176c46541"
server_key_version: "encryptionkey@0"
I20260927 21:34:02.432416 12296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:02.452271 12296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:02.455948 12296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:02.456202 12296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:02.457044 12296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:02.458498 12296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:02.458568 12296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.458705 12296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:02.458753 12296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.490782 12296 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:43201
I20260927 21:34:02.490831 12423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:43201 every 8 connection(s)
I20260927 21:34:02.491875 12296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:34:02.498121 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12296
I20260927 21:34:02.498265 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:34:02.498756 3819 external_mini_cluster.cc:1428] Setting key ac89cef9662036b187596eb57a260c49
Sep 27 21:34:02 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:02.512773 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:02.494150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:40973 (local address 127.3.186.254:44171)
0927 21:34:02.494326 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:02.494332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:02.495398 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 21:34:02.497725 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:02.497730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:02.497792 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:02.497917 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:02.499450 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.500578 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.502948 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.503361 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.503455 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:02.507183 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:02.507191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:02.507199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:02.507240 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:02.509909 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:02.510755 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:02.510758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:02.510760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:02.510878 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:02.511344 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:02.511347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:02.511349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:02.511571 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:02.511674 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:02.512430 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 21:34:02.512572 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:34:02.514226 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:02.494536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:40973)
0927 21:34:02.495232 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:02.495276 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:02.497437 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:02.498128 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:02.498135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:02.498679 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:02.499233 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:02.499293 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.500740 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.500746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:02.502807 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:02.502824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.503584 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.503593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:02.503888 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:02.504568 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:02.504603 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:02.506990 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:02.510026 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:02.510037 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:02.510070 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:02.510644 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:02.510998 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:02.511006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:02.511013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:02.511242 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:02.511686 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:02.511696 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:02.512083 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 21:34:02.512352 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 21:34:02.512873 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":272,"threads_started":1}
I20260927 21:34:02.515506 12424 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44171
I20260927 21:34:02.515767 12424 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:02.516376 12424 heartbeater.cc:511] Master 127.3.186.254:44171 requested a full tablet report, sending...
I20260927 21:34:02.517663 12101 ts_manager.cc:194] Registered new tserver with Master: d6a17b8bf2fa4bc19b9ef4be53709ab3 (127.3.186.194:43201)
I20260927 21:34:02.518376 12101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:40973
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:34:02.544358 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44171
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:02.843798 12429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:02.844062 12429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:02.844214 12429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:02.850766 12429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:02.850900 12429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:02.851078 12429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:34:02.860548 12429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:02.861739 12429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:02.862901 12429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:02.874799 12434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:02.874728 12435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:02.874840 12437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:02.875924 12429 server_base.cc:1034] running on GCE node
I20260927 21:34:02.876459 12429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:02.877295 12429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:02.878471 12429 hybrid_clock.cc:648] HybridClock initialized: now 1790544842878436 us; error 36 us; skew 500 ppm
I20260927 21:34:02.878777 12429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:02 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:02.881764 12429 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:34:02.882963 12429 webserver.cc:460] Webserver started at http://127.3.186.195:38989/ using document root <none> and password file <none>
I20260927 21:34:02.883523 12429 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:02.883613 12429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:02.883841 12429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:02.886411 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "1a403fd6944d43c58eefb33e2f08ac13"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "25c33dd279af42c43b735c5c6dbe413b"
server_key_iv: "4bf3b6b864c03e3072e4a824cfe0bf6a"
server_key_version: "encryptionkey@0"
I20260927 21:34:02.887084 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "1a403fd6944d43c58eefb33e2f08ac13"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "25c33dd279af42c43b735c5c6dbe413b"
server_key_iv: "4bf3b6b864c03e3072e4a824cfe0bf6a"
server_key_version: "encryptionkey@0"
I20260927 21:34:02.891644 12429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 21:34:02.894750 12444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.895915 12429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:34:02.896070 12429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "1a403fd6944d43c58eefb33e2f08ac13"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "25c33dd279af42c43b735c5c6dbe413b"
server_key_iv: "4bf3b6b864c03e3072e4a824cfe0bf6a"
server_key_version: "encryptionkey@0"
I20260927 21:34:02.896245 12429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:02.911051 12429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:02.915030 12429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:02.915290 12429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:02.916054 12429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:02.917177 12429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:02.917270 12429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.917412 12429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:02.917484 12429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:02.947041 12429 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:33201
I20260927 21:34:02.947074 12556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:33201 every 8 connection(s)
I20260927 21:34:02.948143 12429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:34:02.949589 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12429
I20260927 21:34:02.949754 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:34:02.950198 3819 external_mini_cluster.cc:1428] Setting key 0fe917f8538568ee1159767647946b11
Sep 27 21:34:02 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:02.966607 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:02.950676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:36931 (local address 127.3.186.254:44171)
0927 21:34:02.951010 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:02.951015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:02.952056 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 21:34:02.954182 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:02.954187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:02.954259 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:02.954364 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:02.955645 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.956348 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.957877 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.958288 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.958438 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:02.961364 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:02.961369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:02.961372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:02.961400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:02.963662 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:02.964385 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:02.964392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:02.964395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:02.964515 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:02.964987 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:02.964992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:02.964995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:02.965239 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:02.965388 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:02.966080 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 21:34:02.966281 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260927 21:34:02.967398 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:02.951172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:36931)
0927 21:34:02.951922 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:02.951974 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:02.953964 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:02.954564 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:02.954571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:02.955047 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:02.955506 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:02.955520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.956497 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.956507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:02.957746 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:02.957757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:02.958500 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:02.958509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:02.958725 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:02.959224 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:02.959242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:02.961170 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:02.963826 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:02.963834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:02.963847 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:02.964267 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:02.964632 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:02.964638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:02.964641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:02.964876 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:02.965362 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:02.965368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:02.965737 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 21:34:02.965982 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 21:34:02.966417 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
I20260927 21:34:02.968462 12557 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44171
I20260927 21:34:02.968717 12557 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:02.969334 12557 heartbeater.cc:511] Master 127.3.186.254:44171 requested a full tablet report, sending...
I20260927 21:34:02.970718 12101 ts_manager.cc:194] Registered new tserver with Master: 1a403fd6944d43c58eefb33e2f08ac13 (127.3.186.195:33201)
I20260927 21:34:02.971451 12101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:36931
I20260927 21:34:02.976876 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab.
Entry for principal oryx/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab.
WARNING: no policy specified for oryx/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab.
Entry for principal oryx/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab.
I20260927 21:34:03.058166 12290 heartbeater.cc:503] Master 127.3.186.254:44171 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab.
Entry for principal oryx/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab.
I20260927 21:34:03.084270 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12162
Sep 27 21:34:03 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:34:03.103832 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:03.085092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.3.186.193:39319)
0927 21:34:03.085684 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:03.085691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:03.085716 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:34:03.085824 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:03.085830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:03.086036 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:03.086163 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:03.087341 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:03.088269 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:03.089366 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:03.089911 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:03.090009 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:03.099016 (+ 9007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:03.099026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:03.099038 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:03.099066 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:03.101836 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:03.102360 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:03.102366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:03.102368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:03.102475 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:03.102738 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:03.102741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:03.102743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:03.103041 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:03.103248 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:03.103410 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:34:03.103579 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260927 21:34:03.104936 12285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:03.184167 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12162
I20260927 21:34:03.200023 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12296
Sep 27 21:34:03 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:34:03.213991 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:03.200833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.3.186.194:43201)
0927 21:34:03.201386 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:03.201400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:03.201429 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:34:03.201487 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:03.201493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:03.201753 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:03.201880 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:03.202943 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:03.203806 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:03.204915 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:03.205366 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:03.205467 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:03.209156 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:03.209165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:03.209177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:03.209203 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:03.211932 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:03.212434 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:03.212440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:03.212442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:03.212539 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:03.212796 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:03.212798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:03.212800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:03.213063 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:03.213261 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:03.213450 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 21:34:03.213726 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260927 21:34:03.215181 12419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:03.293378 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12296
I20260927 21:34:03.308590 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12429
Sep 27 21:34:03 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:34:03.323195 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:03.309485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.3.186.195:33201)
0927 21:34:03.310073 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:03.310079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:03.310123 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:34:03.310177 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:03.310183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:03.310366 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:03.310487 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:03.311619 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:03.312589 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:03.313779 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:03.314254 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:03.314338 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:03.318185 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:03.318194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:03.318204 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:03.318231 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:03.320825 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:03.321375 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:03.321381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:03.321383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:03.321492 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:03.321970 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:03.321973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:03.321975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:03.322253 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:03.322484 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:03.322719 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 21:34:03.322905 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
I20260927 21:34:03.324281 12552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:03.403720 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12429
I20260927 21:34:03.418766 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12069
I20260927 21:34:03.420197 12131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:03.515373 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12069
I20260927 21:34:03.534868 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44171
--webserver_interface=127.3.186.254
--webserver_port=43983
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:03.839037 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:03.839296 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:03.839395 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:03.846046 12574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:34:03.846125 12574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:03.846167 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:03.846221 12574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:34:03.846256 12574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:34:03.855019 12574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:03.855993 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:03.857002 12574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:03.867550 12579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:03.867619 12580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:03.867576 12582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:03.868858 12574 server_base.cc:1034] running on GCE node
I20260927 21:34:03.869490 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:03.870700 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:03.871884 12574 hybrid_clock.cc:648] HybridClock initialized: now 1790544843871851 us; error 27 us; skew 500 ppm
I20260927 21:34:03.872188 12574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:03 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:03.875064 12574 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:34:03.876331 12574 webserver.cc:460] Webserver started at http://127.3.186.254:43983/ using document root <none> and password file <none>
I20260927 21:34:03.876924 12574 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:03.877069 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:03.881907 12574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:34:03.884374 12589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:03.885612 12574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:34:03.885768 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "176be78961e845a08e466b99afcda311"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "ad168bb7584577f1f9ae61806cb2f745"
server_key_iv: "0835312d8afc421cfc7c02cab4c0c2a3"
server_key_version: "encryptionkey@0"
I20260927 21:34:03.886607 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:03.910568 12574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:03.914325 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:03.914567 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:03.931502 12574 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44171
I20260927 21:34:03.931569 12640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44171 every 8 connection(s)
I20260927 21:34:03.932727 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:34:03.938752 12641 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:34:03.941694 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Bootstrap starting.
I20260927 21:34:03.942829 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12574
I20260927 21:34:03.943801 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:39319
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:44171
--webserver_port=36355
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:34:03.961694 12641 log.cc:824] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:03.969442 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:34:03.969920 12641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Bootstrap complete.
I20260927 21:34:03.978749 12641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:03.979559 12641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176be78961e845a08e466b99afcda311, State: Initialized, Role: FOLLOWER
I20260927 21:34:03.980144 12641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:03.980360 12641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:34:03.980491 12641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:34:03.980669 12641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:34:03.984086 12641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:03.984540 12641 leader_election.cc:302] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 176be78961e845a08e466b99afcda311; no voters:
I20260927 21:34:03.985143 12641 leader_election.cc:288] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:34:03.985262 12645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:34:03.986362 12645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [term 2 LEADER]: Becoming Leader. State: Replica: 176be78961e845a08e466b99afcda311, State: Running, Role: LEADER
I20260927 21:34:03.986903 12645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } }
I20260927 21:34:03.987219 12641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:34:03.989295 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176be78961e845a08e466b99afcda311. Latest consensus state: current_term: 2 leader_uuid: "176be78961e845a08e466b99afcda311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } } }
I20260927 21:34:03.990134 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:03.989652 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "176be78961e845a08e466b99afcda311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176be78961e845a08e466b99afcda311" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44171 } } }
I20260927 21:34:03.990749 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:04.002036 12652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:34:04.006734 12652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:34:04.008061 12652 catalog_manager.cc:1259] Loaded cluster ID: c6a8aceca5d14801a2344561f2d99aee
I20260927 21:34:04.008193 12652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:34:04.012024 12652 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:34:04.013378 12652 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 176be78961e845a08e466b99afcda311: Loaded TSK: 0
I20260927 21:34:04.014792 12652 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:34:04.284050 12643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:04.284278 12643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:04.284407 12643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:04.291419 12643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:04.291512 12643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:04.291715 12643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:34:04.301044 12643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.193.keytab
--principal=oryx/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:39319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=36355
--tserver_master_addrs=127.3.186.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:04.302186 12643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:04.303313 12643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:04.316375 12667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:04.317026 12668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:04.317386 12670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:04.318341 12643 server_base.cc:1034] running on GCE node
I20260927 21:34:04.318928 12643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:04.319842 12643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:04.321049 12643 hybrid_clock.cc:648] HybridClock initialized: now 1790544844320986 us; error 58 us; skew 500 ppm
I20260927 21:34:04.321333 12643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:04 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544844, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:04.324371 12643 init.cc:377] Logged in from keytab as oryx/127.3.186.193@KRBTEST.COM (short username oryx)
I20260927 21:34:04.325850 12643 webserver.cc:460] Webserver started at http://127.3.186.193:36355/ using document root <none> and password file <none>
I20260927 21:34:04.326478 12643 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:04.326619 12643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:04.331528 12643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:34:04.334067 12677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:04.335641 12643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:34:04.335773 12643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "4cbf4434ae2e40fb8ea18a84afc862ff"
format_stamp: "Formatted at 2026-09-27 21:34:01 on dist-test-slave-pqr8"
server_key: "01a1f4301427b79015dd9bf4eecf28bb"
server_key_iv: "a09f0441b83af146932d44d60631af4b"
server_key_version: "encryptionkey@0"
I20260927 21:34:04.336959 12643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:04.350651 12643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:04.354161 12643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:04.354413 12643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:04.355223 12643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:04.356371 12643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:04.356459 12643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:04.356616 12643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:04.356710 12643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:04.387579 12643 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:39319
I20260927 21:34:04.387650 12789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:39319 every 8 connection(s)
I20260927 21:34:04.388965 12643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:34:04.395248 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12643
I20260927 21:34:04.395972 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:43201
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:44171
--webserver_port=45115
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:34:04 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:04 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:34:04.705446 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:04.705781 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:04.705943 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:04.712860 12794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:04.713011 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:04.713162 12794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:34:04.722352 12794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.194.keytab
--principal=oryx/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:43201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=45115
--tserver_master_addrs=127.3.186.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:04.723505 12794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:04.724713 12794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:04.741416 12800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:04.741456 12801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:04.742842 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:04.743191 12794 server_base.cc:1034] running on GCE node
I20260927 21:34:04.744184 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:04.745705 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:04.747088 12794 hybrid_clock.cc:648] HybridClock initialized: now 1790544844747009 us; error 49 us; skew 500 ppm
I20260927 21:34:04.747602 12794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:04 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544844, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:04.752381 12794 init.cc:377] Logged in from keytab as oryx/127.3.186.194@KRBTEST.COM (short username oryx)
I20260927 21:34:04.753978 12794 webserver.cc:460] Webserver started at http://127.3.186.194:45115/ using document root <none> and password file <none>
I20260927 21:34:04.754626 12794 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:04.754755 12794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:04.760953 12794 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:34:04.764007 12810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:04.765699 12794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:34:04.765856 12794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "d6a17b8bf2fa4bc19b9ef4be53709ab3"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "86a3e4d34c0a1c9bad73449f500c2663"
server_key_iv: "559586694ef12b56749052c176c46541"
server_key_version: "encryptionkey@0"
I20260927 21:34:04.766819 12794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:04.789974 12794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:04.794296 12794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:04.794564 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:04.795578 12794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:04.796856 12794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:04.796986 12794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:04.797109 12794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:04.797195 12794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:04.844632 12794 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:43201
I20260927 21:34:04.844686 12922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:43201 every 8 connection(s)
I20260927 21:34:04.846068 12794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:34:04.849718 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12794
I20260927 21:34:04.850628 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:33201
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:44171
--webserver_port=38989
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:34:04 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:04 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:34:05.186615 12927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:05.186875 12927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:05.187053 12927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:05.194397 12927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:05.194525 12927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:05.194694 12927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:34:05.203552 12927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/krb5kdc/oryx_127.3.186.195.keytab
--principal=oryx/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:33201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=38989
--tserver_master_addrs=127.3.186.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:05.204670 12927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:05.205816 12927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:05.222154 12933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:05.222471 12934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:05.227991 12927 server_base.cc:1034] running on GCE node
W20260927 21:34:05.228292 12936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:05.228891 12927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:05.237818 12927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:05.239053 12927 hybrid_clock.cc:648] HybridClock initialized: now 1790544845239008 us; error 53 us; skew 500 ppm
I20260927 21:34:05.239351 12927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544845, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:05.242662 12927 init.cc:377] Logged in from keytab as oryx/127.3.186.195@KRBTEST.COM (short username oryx)
I20260927 21:34:05.244282 12927 webserver.cc:460] Webserver started at http://127.3.186.195:38989/ using document root <none> and password file <none>
I20260927 21:34:05.244896 12927 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:05.245051 12927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:05.250105 12927 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:34:05.252602 12943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:05.253893 12927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:34:05.254061 12927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "1a403fd6944d43c58eefb33e2f08ac13"
format_stamp: "Formatted at 2026-09-27 21:34:02 on dist-test-slave-pqr8"
server_key: "25c33dd279af42c43b735c5c6dbe413b"
server_key_iv: "4bf3b6b864c03e3072e4a824cfe0bf6a"
server_key_version: "encryptionkey@0"
I20260927 21:34:05.254882 12927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:05.279451 12927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:05.283505 12927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:05.283823 12927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:05.285102 12927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:05.286438 12927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:05.286556 12927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:05.286684 12927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:05.286769 12927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:05.334781 12927 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:33201
I20260927 21:34:05.334846 13055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:33201 every 8 connection(s)
I20260927 21:34:05.336126 12927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:34:05.340996 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 12927
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:05.391978 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:05.342327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.3.186.254:44171)
0927 21:34:05.370027 (+ 27700us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:05.370033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:05.370051 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:34:05.370194 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:05.370203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:05.370304 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:05.370430 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:05.373410 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.374991 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.377098 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.377762 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.377891 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.381283 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:05.381295 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:05.381312 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:05.381357 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:05.385450 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:05.387275 (+ 1825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:05.387285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:05.387296 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:05.387574 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:05.388049 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:05.388055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:05.388059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:05.388735 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:05.389001 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:05.390418 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0927 21:34:05.390831 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27595,"thread_start_us":27501,"threads_started":1}
W20260927 21:34:05.923964 12793 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.521s user 0.002s sys 0.015s
W20260927 21:34:05.925928 12792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:04.391732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:60701 (local address 127.3.186.254:44171)
0927 21:34:04.392382 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:04.392396 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:04.392610 (+ 214us) server_negotiation.cc:407] Connection header received
0927 21:34:04.394570 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:04.394581 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:04.394854 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:04.395125 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:04.396967 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.398178 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.400568 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.401692 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.401813 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.925486 (+1523673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:60701: BlockingRecv error: recv got EOF from 127.3.186.193:60701 (error 108)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":229,"threads_started":1}
I20260927 21:34:05.926700 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:04.391765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:60701)
0927 21:34:04.392466 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:04.392505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:04.394116 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:04.395333 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:04.395344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:04.396038 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:04.396789 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:04.396815 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.398333 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.398341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:04.400397 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:04.400423 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.401894 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.401903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:04.402206 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:04.402872 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:04.402902 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:05.925107 (+1522205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":330,"threads_started":1}
W20260927 21:34:05.928574 12790 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44171 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.186.254:44171: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:34:05.944283 12792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:05.929570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:56665 (local address 127.3.186.254:44171)
0927 21:34:05.929785 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:05.929794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:05.929934 (+ 140us) server_negotiation.cc:407] Connection header received
0927 21:34:05.930131 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:05.930139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:05.930245 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:05.930402 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:05.931651 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.933253 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.935861 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.936610 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.936730 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.944042 (+ 7312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:56665: BlockingRecv error: recv got EOF from 127.3.186.193:56665 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:34:05.944924 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:05.929457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:56665)
0927 21:34:05.929804 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:05.929829 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:05.929983 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:05.930466 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:05.930474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:05.931095 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:05.931395 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:05.931411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.933574 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.933583 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:05.935638 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:05.935659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.937054 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.937060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:05.937266 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.939854 (+ 2588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:05.939875 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:05.944687 (+ 4812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183}
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:05 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:05.954689 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:05.945654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:39279)
0927 21:34:05.946561 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:05.946575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:05.946706 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:05.946994 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:05.946999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:05.947405 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:05.947618 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:05.947625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.948562 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.948567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:05.949425 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:05.949437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.950114 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.950119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:05.950275 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.950825 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:05.950841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:05.954412 (+ 3571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":624,"threads_started":1}
W20260927 21:34:05.954811 12792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:05.945728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:39279 (local address 127.3.186.254:44171)
0927 21:34:05.945901 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:05.945909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:05.946652 (+ 743us) server_negotiation.cc:407] Connection header received
0927 21:34:05.946808 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:05.946812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:05.946869 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:05.946975 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:05.947745 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.948425 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.949583 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.949954 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.950077 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.954577 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:39279: BlockingRecv error: recv got EOF from 127.3.186.193:39279 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 21:34:05.955119 12790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:34:06.361477 12926 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.484s user 0.003s sys 0.013s
I20260927 21:34:06.362968 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:04.850203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:36047)
0927 21:34:04.863963 (+ 13760us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:04.864030 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:04.867650 (+ 3620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:04.868909 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:04.868921 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:04.869761 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:04.870601 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:04.870630 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.872375 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.872382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:04.874667 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:04.874683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.876081 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.876092 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:04.876436 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:04.877374 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:04.877411 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.362257 (+1484846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13272,"thread_start_us":17861,"threads_started":1}
W20260927 21:34:06.363189 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:04.849159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:36047 (local address 127.3.186.254:44171)
0927 21:34:04.864292 (+ 15133us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:04.864300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:04.864325 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:34:04.868178 (+ 3853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:04.868187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:04.868277 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:04.868407 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:04.870821 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.872146 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.874853 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:04.875806 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:04.876009 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.362648 (+1486639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:36047: BlockingRecv error: recv got EOF from 127.3.186.194:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":14972,"thread_start_us":15124,"threads_started":1}
W20260927 21:34:06.364212 12923 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44171 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.186.254:44171: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:06.373967 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:06.364982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:32841)
0927 21:34:06.365160 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:06.365177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:06.365280 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:06.365615 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:06.365620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:06.365963 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:06.366108 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.366116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.367078 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.367083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.368310 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.368321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.369205 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.369215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.369494 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.370342 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:06.370360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.373711 (+ 3351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:34:06.374104 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:06.365072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:32841 (local address 127.3.186.254:44171)
0927 21:34:06.365250 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:06.365256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:06.365272 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:34:06.365401 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:06.365406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:06.365469 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:06.365595 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:06.366242 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.366929 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.368444 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.368983 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.369109 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.373876 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:32841: BlockingRecv error: recv got EOF from 127.3.186.194:32841 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:34:06.385883 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:06.374703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:41975 (local address 127.3.186.254:44171)
0927 21:34:06.374833 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:06.374836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:06.374855 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:34:06.375055 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:06.375065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:06.375121 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:06.375191 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:06.376312 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.376970 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.379506 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.379900 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.379995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.385724 (+ 5729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:41975: BlockingRecv error: recv got EOF from 127.3.186.194:41975 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:34:06.385967 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:06.374555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:41975)
0927 21:34:06.374781 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:06.374805 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:06.374935 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:06.375272 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:06.375291 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:06.375851 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:06.376121 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.376135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.377123 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.377131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.379321 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.379340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.380063 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.380072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.380380 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.381138 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:06.381170 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.385610 (+ 4440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 21:34:06.386364 12923 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:34:06.542884 13060 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.158s user 0.005s sys 0.008s
W20260927 21:34:06.544034 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:05.369997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:51795 (local address 127.3.186.254:44171)
0927 21:34:05.376422 (+ 6425us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:05.376427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:05.376444 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:34:05.376484 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:05.376488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:05.376545 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:05.378631 (+ 2086us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:05.378827 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.380080 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.382564 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.383286 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.384162 (+ 876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.543809 (+1159647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:51795: BlockingRecv error: recv got EOF from 127.3.186.195:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":6316,"thread_start_us":6649,"threads_started":1}
I20260927 21:34:06.544664 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:05.350502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:51795)
0927 21:34:05.364227 (+ 13725us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:05.364278 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:05.367412 (+ 3134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:05.376998 (+ 9586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:05.377012 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:05.377720 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:05.378464 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:05.378490 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.380275 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.380283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:05.382372 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:05.382387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:05.383528 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:05.383536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:05.383821 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:05.384420 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:05.384445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.543658 (+1159213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13349,"thread_start_us":17223,"threads_started":1}
W20260927 21:34:06.545595 13056 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44171 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.186.254:44171: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:06.554108 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:06.546204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:50925)
0927 21:34:06.546451 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:06.546467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:06.546555 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:06.546907 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:06.546911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:06.547239 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:06.547378 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.547385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.548367 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.548374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.549282 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.549291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.550025 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.550030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.550177 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.550775 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:06.550792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.553849 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 21:34:06.554216 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:06.546268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:50925 (local address 127.3.186.254:44171)
0927 21:34:06.546695 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:06.546698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:06.546710 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:34:06.546743 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:06.546746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:06.546792 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:06.546885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:06.547549 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.548242 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.549408 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.549876 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.550000 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.554011 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:50925: BlockingRecv error: recv got EOF from 127.3.186.195:50925 (error 108)
Metrics: {"server-negotiator.queue_time_us":349}
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:06.562289 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:06.554773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:47339)
0927 21:34:06.554928 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:06.554966 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:06.555036 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:06.555304 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:06.555307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:06.555566 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:06.555706 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.555713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.556560 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.556564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.557802 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.557816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.558646 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.558652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.558840 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.559401 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:06.559415 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.562130 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:34:06.562531 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:06.554830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:47339 (local address 127.3.186.254:44171)
0927 21:34:06.555059 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:06.555063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:06.555097 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:34:06.555132 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:06.555135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:06.555192 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:06.555264 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:06.555819 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.556440 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.557935 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.558482 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.558619 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.562276 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:47339: BlockingRecv error: recv got EOF from 127.3.186.195:47339 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260927 21:34:06.562548 13056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:06 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:06.963891 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:06.955719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:42945)
0927 21:34:06.956165 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:06.956181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:06.956282 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:06.956558 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:06.956560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:06.956886 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:06.957025 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.957032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.958038 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.958042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.959048 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:06.959060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.959681 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.959685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:06.959789 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.960389 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:06.960405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:06.963695 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":245,"threads_started":1}
W20260927 21:34:06.964059 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:06.955757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:42945 (local address 127.3.186.254:44171)
0927 21:34:06.955917 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:06.955921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:06.956250 (+ 329us) server_negotiation.cc:407] Connection header received
0927 21:34:06.956372 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:06.956375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:06.956471 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:06.956546 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:06.957134 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.957911 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.959167 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:06.959561 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:06.959650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:06.963855 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:42945: BlockingRecv error: recv got EOF from 127.3.186.193:42945 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:34:07 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:07 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:07.396201 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:07.386926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:38221)
0927 21:34:07.387422 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:07.387438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:07.387613 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:07.387987 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:07.387992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:07.388441 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:07.388630 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:07.388645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.390052 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.390058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:07.391076 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:07.391088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.391858 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.391864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:07.392034 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:07.392609 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:07.392625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:07.395973 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
W20260927 21:34:07.396359 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:07.387065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:38221 (local address 127.3.186.254:44171)
0927 21:34:07.387260 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:07.387266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:07.387550 (+ 284us) server_negotiation.cc:407] Connection header received
0927 21:34:07.387772 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:07.387778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:07.387847 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:07.387945 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:07.388782 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.389926 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.391235 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.391680 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.391807 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:07.396147 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:38221: BlockingRecv error: recv got EOF from 127.3.186.194:38221 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 21:34:07 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:07 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:07.571410 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:07.563121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:48575)
0927 21:34:07.563678 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:07.563693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:07.563831 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:07.564203 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:07.564206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:07.564576 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:07.564703 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:07.564711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.565624 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.565627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:07.566548 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:07.566558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.567195 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.567199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:07.567309 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:07.567841 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:07.567857 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:07.571089 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":334,"threads_started":1}
W20260927 21:34:07.571516 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:07.563140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:48575 (local address 127.3.186.254:44171)
0927 21:34:07.563304 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:07.563308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:07.563772 (+ 464us) server_negotiation.cc:407] Connection header received
0927 21:34:07.563984 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:07.563991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:07.564071 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:07.564166 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:07.564817 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.565507 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.566661 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.567056 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.567141 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:07.571260 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:48575: BlockingRecv error: recv got EOF from 127.3.186.195:48575 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 21:34:07 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:07 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:07.973413 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:07.964694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:40279)
0927 21:34:07.965150 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:07.965166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:07.965273 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:07.965607 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:07.965612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:07.966018 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:07.966159 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:07.966167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.967091 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.967095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:07.968042 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:07.968053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.968792 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.968801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:07.968935 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:07.969586 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:07.969604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:07.973048 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
W20260927 21:34:07.973495 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:07.964976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:40279 (local address 127.3.186.254:44171)
0927 21:34:07.965168 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:07.965171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:07.965217 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:34:07.965365 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:07.965369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:07.965423 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:07.965496 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:07.966280 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.966950 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.968192 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:07.968636 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:07.968720 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:07.973256 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:40279: BlockingRecv error: recv got EOF from 127.3.186.193:40279 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 21:34:08 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:08 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:08.409058 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:08.397149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:34959)
0927 21:34:08.397646 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:08.397660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:08.397803 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:08.398087 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:08.398090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:08.398447 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:08.398610 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:08.398618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.399624 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.399628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:08.400507 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:08.400517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.401151 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.401155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:08.401273 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:08.405267 (+ 3994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:08.405284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:08.408715 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
W20260927 21:34:08.409142 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:08.397194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:34959 (local address 127.3.186.254:44171)
0927 21:34:08.397377 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:08.397381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:08.397758 (+ 377us) server_negotiation.cc:407] Connection header received
0927 21:34:08.397882 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:08.397904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:08.397959 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:08.398038 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:08.398736 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.399481 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.400633 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.401011 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.401109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:08.408904 (+ 7795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:34959: BlockingRecv error: recv got EOF from 127.3.186.194:34959 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 21:34:08 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:08 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:08.579636 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:08.572236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:55225)
0927 21:34:08.572747 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:08.572761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:08.572873 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:08.573148 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:08.573150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:08.573530 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:08.573692 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:08.573700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.574608 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.574612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:08.575419 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:08.575428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.575992 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.575996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:08.576095 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:08.576576 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:08.576591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:08.579382 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":297,"threads_started":1}
W20260927 21:34:08.579792 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:08.572359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:55225 (local address 127.3.186.254:44171)
0927 21:34:08.572516 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:08.572520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:08.572831 (+ 311us) server_negotiation.cc:407] Connection header received
0927 21:34:08.572977 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:08.572981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:08.573057 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:08.573134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:08.573802 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.574486 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.575531 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.575885 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.575962 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:08.579571 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:55225: BlockingRecv error: recv got EOF from 127.3.186.195:55225 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 21:34:08 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:08 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:08.982388 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:08.974340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:45543)
0927 21:34:08.974768 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:08.974789 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:08.974931 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:08.975275 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:08.975281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:08.975748 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:08.975914 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:08.975923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.976869 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.976873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:08.977825 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:08.977837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.978526 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.978529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:08.978653 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:08.979183 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:08.979199 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:08.982164 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":218,"threads_started":1}
W20260927 21:34:08.982545 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:08.974387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:45543 (local address 127.3.186.254:44171)
0927 21:34:08.974588 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:08.974595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:08.974866 (+ 271us) server_negotiation.cc:407] Connection header received
0927 21:34:08.975042 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:08.975046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:08.975120 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:08.975213 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:08.976034 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.976751 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.977967 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:08.978404 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:08.978525 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:08.982330 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:45543: BlockingRecv error: recv got EOF from 127.3.186.193:45543 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 21:34:09 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:09 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:09.418315 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:09.410013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:55619)
0927 21:34:09.410473 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:09.410489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:09.410620 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:09.410888 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:09.410891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:09.411204 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:09.411365 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:09.411373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.412318 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.412322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:09.413215 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:09.413226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.413875 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.413879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:09.413997 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:09.414659 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:09.414706 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:09.418097 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
W20260927 21:34:09.418612 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:09.410095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:55619 (local address 127.3.186.254:44171)
0927 21:34:09.410261 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:09.410266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:09.410581 (+ 315us) server_negotiation.cc:407] Connection header received
0927 21:34:09.410713 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:09.410716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:09.410771 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:09.410847 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:09.411484 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.412188 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.413331 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.413760 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.413848 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:09.418314 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:55619: BlockingRecv error: recv got EOF from 127.3.186.194:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 21:34:09 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:09 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:09.589159 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:09.580520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:48517)
0927 21:34:09.581038 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:09.581053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:09.581206 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:09.581590 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:09.581595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:09.581938 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:09.582090 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:09.582097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.583034 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.583041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:09.583860 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:09.583870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.584556 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.584562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:09.584729 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:09.585238 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:09.585254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:09.588833 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":265,"threads_started":1}
W20260927 21:34:09.589289 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:09.580668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:48517 (local address 127.3.186.254:44171)
0927 21:34:09.580891 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:09.580898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:09.581139 (+ 241us) server_negotiation.cc:407] Connection header received
0927 21:34:09.581328 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:09.581333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:09.581405 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:09.581505 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:09.582231 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.582903 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.584010 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.584396 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.584539 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:09.589018 (+ 4479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:48517: BlockingRecv error: recv got EOF from 127.3.186.195:48517 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 21:34:09 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:09 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:09.991451 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:09.983249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:33701)
0927 21:34:09.983672 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:09.983687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:09.983784 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:09.984048 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:09.984051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:09.984410 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:09.984547 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:09.984553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.985561 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.985566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:09.986524 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:09.986534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.987255 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.987258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:09.987368 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:09.988032 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:09.988055 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:09.991214 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":229,"threads_started":1}
W20260927 21:34:09.991585 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:09.983281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:33701 (local address 127.3.186.254:44171)
0927 21:34:09.983489 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:09.983498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:09.983761 (+ 263us) server_negotiation.cc:407] Connection header received
0927 21:34:09.983874 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:09.983878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:09.983933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:09.984008 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:09.984691 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.985403 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.986647 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:09.987141 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:09.987229 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:09.991375 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:33701: BlockingRecv error: recv got EOF from 127.3.186.193:33701 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 21:34:10 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:10 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:10.427632 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:10.419257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:44535)
0927 21:34:10.419655 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:10.419684 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:10.419786 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:10.420089 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:10.420092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:10.420422 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:10.420576 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:10.420584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.421539 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.421543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:10.422703 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:10.422714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.423478 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.423483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:10.423648 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:10.424175 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:10.424191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:10.427247 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":221,"threads_started":1}
W20260927 21:34:10.427731 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:10.419342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:44535 (local address 127.3.186.254:44171)
0927 21:34:10.419562 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:10.419568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:10.419758 (+ 190us) server_negotiation.cc:407] Connection header received
0927 21:34:10.419886 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:10.419893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:10.419970 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:10.420044 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:10.420721 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.421421 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.422836 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.423336 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.423459 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:10.427459 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:44535: BlockingRecv error: recv got EOF from 127.3.186.194:44535 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 21:34:10 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:10 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:10.598488 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:10.590163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:33727)
0927 21:34:10.590692 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:10.590707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:10.590846 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:10.591118 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:10.591121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:10.591477 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:10.591618 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:10.591626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.592512 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.592515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:10.593452 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:10.593461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.594113 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.594117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:10.594237 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:10.594852 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:10.594868 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:10.598262 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":281,"threads_started":1}
W20260927 21:34:10.598680 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:10.590190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:33727 (local address 127.3.186.254:44171)
0927 21:34:10.590398 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:10.590407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:10.590807 (+ 400us) server_negotiation.cc:407] Connection header received
0927 21:34:10.590945 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:10.590948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:10.591004 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:10.591081 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:10.591729 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.592409 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.593590 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.593998 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.594086 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:10.598417 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:33727: BlockingRecv error: recv got EOF from 127.3.186.195:33727 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 21:34:10 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:10 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:10.999853 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:10.992425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:55613)
0927 21:34:10.992832 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:10.992846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:10.992942 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:10.993199 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:10.993201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:10.993538 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:10.993677 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:10.993684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.994591 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.994596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:10.995467 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:10.995476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.996112 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.996116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:10.996231 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:10.996739 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:10.996755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:10.999665 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":227,"threads_started":1}
W20260927 21:34:11.000038 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:10.992511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:55613 (local address 127.3.186.254:44171)
0927 21:34:10.992723 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:10.992729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:10.992915 (+ 186us) server_negotiation.cc:407] Connection header received
0927 21:34:10.993028 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:10.993031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:10.993091 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:10.993161 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:10.993783 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.994476 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.995598 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:10.995982 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:10.996076 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:10.999844 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:55613: BlockingRecv error: recv got EOF from 127.3.186.193:55613 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 21:34:11 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:11 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:11.437166 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:11.428514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:50029)
0927 21:34:11.428977 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:11.428992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:11.429130 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:11.429458 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:11.429463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:11.429905 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:11.430041 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:11.430049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.431167 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.431173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:11.432292 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:11.432303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.432960 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.432964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:11.433067 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:11.433609 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:11.433624 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:11.436953 (+ 3329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":263,"threads_started":1}
W20260927 21:34:11.437371 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:11.428702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:50029 (local address 127.3.186.254:44171)
0927 21:34:11.428900 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:11.428905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:11.429064 (+ 159us) server_negotiation.cc:407] Connection header received
0927 21:34:11.429261 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:11.429266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:11.429334 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:11.429433 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:11.430181 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.430975 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.432448 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.432851 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.432938 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:11.437110 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:50029: BlockingRecv error: recv got EOF from 127.3.186.194:50029 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 21:34:11 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:11 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:11.607697 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:11.599315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:32829)
0927 21:34:11.599780 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:11.599795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:11.599900 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:11.600217 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:11.600220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:11.600572 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:11.600737 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:11.600744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.601700 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.601705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:11.602574 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:11.602583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.603235 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.603239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:11.603362 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:11.603966 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:11.603982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:11.607466 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":283,"threads_started":1}
W20260927 21:34:11.607848 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:11.599399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:32829 (local address 127.3.186.254:44171)
0927 21:34:11.599617 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:11.599623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:11.599872 (+ 249us) server_negotiation.cc:407] Connection header received
0927 21:34:11.600009 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:11.600013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:11.600089 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:11.600166 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:11.600869 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.601579 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.602695 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:11.603097 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:11.603186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:11.607629 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:32829: BlockingRecv error: recv got EOF from 127.3.186.195:32829 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 21:34:12 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:12 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:12.008497 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:12.000590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:50155)
0927 21:34:12.001081 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:12.001099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:12.001212 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:12.001478 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:12.001481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:12.001933 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:12.002074 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:12.002082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.002973 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.002976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:12.003871 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:12.003880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.004542 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.004546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:12.004696 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:12.005205 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:12.005221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:12.008289 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":285,"threads_started":1}
W20260927 21:34:12.008661 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:12.000696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:50155 (local address 127.3.186.254:44171)
0927 21:34:12.000912 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:12.000916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:12.001183 (+ 267us) server_negotiation.cc:407] Connection header received
0927 21:34:12.001304 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:12.001308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:12.001362 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:12.001437 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:12.002191 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.002863 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.003990 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.004425 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.004511 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:12.008451 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:50155: BlockingRecv error: recv got EOF from 127.3.186.193:50155 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 21:34:12 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:12 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:12.446161 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:12.438093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:32827)
0927 21:34:12.438547 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:12.438562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:12.438669 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:12.438963 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:12.438965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:12.439322 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:12.439471 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:12.439479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.440488 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.440492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:12.441449 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:12.441460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.442320 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.442324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:12.442436 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:12.442965 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:12.442980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:12.445953 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
W20260927 21:34:12.446341 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:12.438102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:32827 (local address 127.3.186.254:44171)
0927 21:34:12.438299 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:12.438308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:12.438639 (+ 331us) server_negotiation.cc:407] Connection header received
0927 21:34:12.438773 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:12.438777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:12.438847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:12.438925 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:12.439586 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.440359 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.441738 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.442194 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.442285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:12.446121 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:32827: BlockingRecv error: recv got EOF from 127.3.186.194:32827 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 21:34:12 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:12 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:12.616915 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:12.608544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:47331)
0927 21:34:12.608974 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:12.608989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:12.609094 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:12.609351 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:12.609354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:12.609776 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:12.609937 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:12.609947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.610988 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.610995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:12.611921 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:12.611942 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.612653 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.612658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:12.612838 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:12.613382 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:12.613398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:12.616588 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":242,"threads_started":1}
W20260927 21:34:12.617007 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:12.608706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:47331 (local address 127.3.186.254:44171)
0927 21:34:12.608912 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:12.608918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:12.609062 (+ 144us) server_negotiation.cc:407] Connection header received
0927 21:34:12.609176 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:12.609180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:12.609234 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:12.609306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:12.610074 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.610860 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.612065 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:12.612497 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:12.612624 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:12.616773 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:47331: BlockingRecv error: recv got EOF from 127.3.186.195:47331 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 21:34:13 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:13 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:13.017446 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:13.009314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:38473)
0927 21:34:13.009831 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:13.009847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:13.009950 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:13.010222 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:13.010225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:13.010573 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:13.010750 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:13.010757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.011690 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.011694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:13.012612 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:13.012622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.013301 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.013305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:13.013408 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:13.013957 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:13.013973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:13.017208 (+ 3235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":254,"threads_started":1}
W20260927 21:34:13.017633 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:13.009406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:38473 (local address 127.3.186.254:44171)
0927 21:34:13.009651 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:13.009657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:13.009922 (+ 265us) server_negotiation.cc:407] Connection header received
0927 21:34:13.010049 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:13.010053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:13.010108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:13.010184 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:13.010886 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.011572 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.012747 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.013181 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.013270 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:13.017397 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:38473: BlockingRecv error: recv got EOF from 127.3.186.193:38473 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 21:34:13 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:13 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:13.460394 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:13.446995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:49953)
0927 21:34:13.447513 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:13.447527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:13.447700 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:13.448016 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:13.448020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:13.448417 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:13.448595 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:13.448603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.449600 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.449606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:13.450521 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:13.450532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.451282 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.451287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:13.451449 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:13.456896 (+ 5447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:13.456912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:13.460148 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":251,"threads_started":1}
W20260927 21:34:13.460537 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:13.447077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:49953 (local address 127.3.186.254:44171)
0927 21:34:13.447272 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:13.447280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:13.447608 (+ 328us) server_negotiation.cc:407] Connection header received
0927 21:34:13.447818 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:13.447825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:13.447886 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:13.447985 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:13.448771 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.449446 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.450657 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.451133 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.451272 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:13.460322 (+ 9050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:49953: BlockingRecv error: recv got EOF from 127.3.186.194:49953 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:34:13 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:13 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:13.625766 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:13.617889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:43415)
0927 21:34:13.618507 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:13.618522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:13.618623 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:13.618894 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:13.618899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:13.619265 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:13.619420 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:13.619428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.620371 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.620375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:13.621305 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:13.621315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.621978 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.621982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:13.622097 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:13.622592 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:13.622606 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:13.625486 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":298,"threads_started":1}
W20260927 21:34:13.625846 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:13.617904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:43415 (local address 127.3.186.254:44171)
0927 21:34:13.618092 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:13.618099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:13.618593 (+ 494us) server_negotiation.cc:407] Connection header received
0927 21:34:13.618706 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:13.618710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:13.618764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:13.618837 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:13.619534 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.620230 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.621423 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:13.621858 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:13.621947 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:13.625695 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:43415: BlockingRecv error: recv got EOF from 127.3.186.195:43415 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 21:34:14 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:14 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:14.026952 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:14.018430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:35425)
0927 21:34:14.018901 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:14.018918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:14.019020 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:14.019268 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:14.019270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:14.019606 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:14.019788 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:14.019796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.020708 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.020711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:14.022042 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:14.022056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.022893 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.022898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:14.023008 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:14.023547 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:14.023562 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:14.026740 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
W20260927 21:34:14.027132 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:14.018569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:35425 (local address 127.3.186.254:44171)
0927 21:34:14.018787 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:14.018792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:14.018983 (+ 191us) server_negotiation.cc:407] Connection header received
0927 21:34:14.019094 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:14.019097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:14.019149 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:14.019221 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:14.019893 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.020561 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.022174 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.022689 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.022801 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:14.026915 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:35425: BlockingRecv error: recv got EOF from 127.3.186.193:35425 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:34:14 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:14 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:14.469137 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:14.461397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:36589)
0927 21:34:14.461851 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:14.461866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:14.461969 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:14.462278 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:14.462281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:14.462620 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:14.462768 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:14.462775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.463879 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.463882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:14.464749 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:14.464758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.465388 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.465392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:14.465538 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:14.466051 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:14.466066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:14.468936 (+ 2870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":246,"threads_started":1}
W20260927 21:34:14.469326 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:14.461487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:36589 (local address 127.3.186.254:44171)
0927 21:34:14.461737 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:14.461744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:14.461945 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:34:14.462088 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:14.462104 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:14.462157 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:14.462238 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:14.462939 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.463765 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.464865 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.465253 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.465345 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:14.469122 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:36589: BlockingRecv error: recv got EOF from 127.3.186.194:36589 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 27 21:34:14 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:14 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:14.635310 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:14.626624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:41285)
0927 21:34:14.627109 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:14.627124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:14.627276 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:14.627585 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:14.627589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:14.628013 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:14.628177 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:14.628185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.629175 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.629182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:14.630110 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:14.630120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.630827 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.630833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:14.631009 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:14.631624 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:14.631640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:14.634976 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
W20260927 21:34:14.635367 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:14.626704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:41285 (local address 127.3.186.254:44171)
0927 21:34:14.626938 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:14.626944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:14.627206 (+ 262us) server_negotiation.cc:407] Connection header received
0927 21:34:14.627391 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:14.627396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:14.627458 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:14.627555 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:14.628324 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.629047 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.630242 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:14.630661 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:14.630802 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:14.635142 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:41285: BlockingRecv error: recv got EOF from 127.3.186.195:41285 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 21:34:15 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:15 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:15.036299 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:15.027924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:35255)
0927 21:34:15.028396 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:15.028411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:15.028549 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:15.028875 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:15.028879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:15.029280 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:15.029413 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:15.029422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.030403 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.030409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:15.031352 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:15.031362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.032052 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.032058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:15.032245 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:15.032865 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:15.032881 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:15.035987 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":251,"threads_started":1}
W20260927 21:34:15.036350 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:15.027949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:35255 (local address 127.3.186.254:44171)
0927 21:34:15.028163 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:15.028171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:15.028494 (+ 323us) server_negotiation.cc:407] Connection header received
0927 21:34:15.028674 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:15.028689 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:15.028746 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:15.028845 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:15.029568 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.030274 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.031489 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.031900 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.032027 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:15.036146 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:35255: BlockingRecv error: recv got EOF from 127.3.186.193:35255 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 21:34:15 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:15 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:15.478670 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:15.470104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:33109)
0927 21:34:15.470568 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:15.470582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:15.470721 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:15.471070 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:15.471075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:15.471494 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:15.471630 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:15.471638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.472641 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.472646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:15.473532 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:15.473542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.474169 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.474174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:15.474344 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:15.474901 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:15.474917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:15.478380 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
W20260927 21:34:15.478751 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:15.470153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:33109 (local address 127.3.186.254:44171)
0927 21:34:15.470353 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:15.470361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:15.470662 (+ 301us) server_negotiation.cc:407] Connection header received
0927 21:34:15.470844 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:15.470849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:15.470915 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:15.471034 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:15.471793 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.472517 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.473654 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.474034 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.474159 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:15.478531 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:33109: BlockingRecv error: recv got EOF from 127.3.186.194:33109 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 21:34:15 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:15 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:15.645499 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:15.636225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:47581)
0927 21:34:15.636851 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:15.636867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:15.637012 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:15.637349 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:15.637354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:15.637851 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:15.637981 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:15.637989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.638986 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.638991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:15.639961 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:15.639971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.640683 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.640688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:15.640869 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:15.641499 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:15.641572 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:15.645199 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
W20260927 21:34:15.645607 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:15.636247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:47581 (local address 127.3.186.254:44171)
0927 21:34:15.636459 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:15.636469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:15.636957 (+ 488us) server_negotiation.cc:407] Connection header received
0927 21:34:15.637139 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:15.637144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:15.637213 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:15.637325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:15.638126 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.638845 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.640090 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:15.640534 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:15.640656 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:15.645386 (+ 4730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:47581: BlockingRecv error: recv got EOF from 127.3.186.195:47581 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 21:34:16 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:16 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:16.045925 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:16.037164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:36273)
0927 21:34:16.037627 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:16.037642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:16.037746 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:16.038009 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:16.038011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:16.038320 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:16.038479 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:16.038487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.039441 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.039444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:16.040415 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:16.040451 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.041215 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.041220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:16.041336 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:16.041919 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:16.041935 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:16.045564 (+ 3629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
W20260927 21:34:16.046062 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:16.037321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:36273 (local address 127.3.186.254:44171)
0927 21:34:16.037552 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:16.037560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:16.037712 (+ 152us) server_negotiation.cc:407] Connection header received
0927 21:34:16.037838 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:16.037841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:16.037893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:16.037966 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:16.038597 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.039324 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.040591 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.041084 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.041185 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:16.045802 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:36273: BlockingRecv error: recv got EOF from 127.3.186.193:36273 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 21:34:16 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:16 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:16.488148 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:16.479467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:60533)
0927 21:34:16.479931 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:16.479946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:16.480052 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:16.480322 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:16.480325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:16.480666 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:16.480821 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:16.480829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.481882 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.481891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:16.483086 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:16.483098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.483737 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.483743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:16.483900 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:16.484427 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:16.484443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:16.487826 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":250,"threads_started":1}
W20260927 21:34:16.488299 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:16.479512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:60533 (local address 127.3.186.254:44171)
0927 21:34:16.479708 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:16.479716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:16.480025 (+ 309us) server_negotiation.cc:407] Connection header received
0927 21:34:16.480144 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:16.480148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:16.480203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:16.480276 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:16.480936 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.481710 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.483212 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.483597 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.483733 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:16.488033 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:60533: BlockingRecv error: recv got EOF from 127.3.186.194:60533 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 21:34:16 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:16 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:16.654109 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:16.646315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:57791)
0927 21:34:16.646733 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:16.646748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:16.646852 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:16.647095 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:16.647098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:16.647420 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:16.647555 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:16.647561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.648524 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.648528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:16.649478 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:16.649488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.650158 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.650161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:16.650259 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:16.650826 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:16.650841 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:16.653917 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":227,"threads_started":1}
W20260927 21:34:16.654337 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:16.646423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:57791 (local address 127.3.186.254:44171)
0927 21:34:16.646658 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:16.646664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:16.646814 (+ 150us) server_negotiation.cc:407] Connection header received
0927 21:34:16.646934 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:16.646937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:16.646988 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:16.647059 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:16.647671 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.648406 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.649639 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:16.650046 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:16.650133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:16.654080 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:57791: BlockingRecv error: recv got EOF from 127.3.186.195:57791 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 27 21:34:17 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:17 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:17.055101 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:17.046712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:34987)
0927 21:34:17.047272 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:17.047287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:17.047419 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:17.047753 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:17.047757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:17.048305 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:17.048484 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:17.048493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.049419 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.049422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:17.050414 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:17.050425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.051113 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.051117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:17.051220 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:17.051749 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:17.051765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:17.054902 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":337,"threads_started":1}
W20260927 21:34:17.055254 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:17.046753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:34987 (local address 127.3.186.254:44171)
0927 21:34:17.046941 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:17.046945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:17.047363 (+ 418us) server_negotiation.cc:407] Connection header received
0927 21:34:17.047531 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:17.047536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:17.047603 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:17.047671 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:17.048604 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.049278 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.050545 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.050963 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.051074 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:17.055060 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:34987: BlockingRecv error: recv got EOF from 127.3.186.193:34987 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 21:34:17 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:17 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:17.497422 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:17.489037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:38545)
0927 21:34:17.489566 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:17.489579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:17.489723 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:17.490018 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:17.490023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:17.490417 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:17.490577 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:17.490584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.491559 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.491566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:17.492465 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:17.492475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.493202 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.493207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:17.493378 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:17.493995 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:17.494010 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:17.497208 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
W20260927 21:34:17.497589 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:17.489082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:38545 (local address 127.3.186.254:44171)
0927 21:34:17.489364 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:17.489371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:17.489660 (+ 289us) server_negotiation.cc:407] Connection header received
0927 21:34:17.489838 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:17.489846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:17.489899 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:17.489990 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:17.490704 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.491433 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.492600 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.493056 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.493194 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:17.497361 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:38545: BlockingRecv error: recv got EOF from 127.3.186.194:38545 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Sep 27 21:34:17 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:17 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:17.663674 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:17.654925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:40459)
0927 21:34:17.655340 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:17.655354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:17.655490 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:17.655826 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:17.655830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:17.656240 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:17.656410 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:17.656418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.657311 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.657316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:17.658373 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:17.658384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.659141 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.659147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:17.659313 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:17.659924 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:17.659939 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:17.663471 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
W20260927 21:34:17.663818 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:17.654991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:40459 (local address 127.3.186.254:44171)
0927 21:34:17.655200 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:17.655206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:17.655428 (+ 222us) server_negotiation.cc:407] Connection header received
0927 21:34:17.655628 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:17.655633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:17.655697 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:17.655801 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:17.656537 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.657184 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.658501 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:17.658983 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:17.659109 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:17.663622 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:40459: BlockingRecv error: recv got EOF from 127.3.186.195:40459 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 21:34:18 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:18 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:18.064656 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:18.056045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:58721)
0927 21:34:18.056491 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:18.056505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:18.056649 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:18.056965 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:18.056969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:18.057355 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:18.057535 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:18.057544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.058587 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.058594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:18.059550 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:18.059561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.060321 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.060326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:18.060500 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:18.061021 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:18.061041 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:18.064326 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
W20260927 21:34:18.064779 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:18.056071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:58721 (local address 127.3.186.254:44171)
0927 21:34:18.056302 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:18.056309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:18.056589 (+ 280us) server_negotiation.cc:407] Connection header received
0927 21:34:18.056761 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:18.056766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:18.056830 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:18.056933 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:18.057686 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.058457 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.059692 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.060183 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.060310 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:18.064494 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:58721: BlockingRecv error: recv got EOF from 127.3.186.193:58721 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 21:34:18 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:18 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:18.512197 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:18.498303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:51533)
0927 21:34:18.498748 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:18.498783 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:18.498891 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:18.499167 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:18.499170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:18.499505 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:18.499666 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:18.499674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.500564 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.500568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:18.501471 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:18.501481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.502126 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.502130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:18.502248 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:18.508879 (+ 6631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:18.508896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:18.511964 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
W20260927 21:34:18.512358 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:18.498338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:51533 (local address 127.3.186.254:44171)
0927 21:34:18.498551 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:18.498560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:18.498861 (+ 301us) server_negotiation.cc:407] Connection header received
0927 21:34:18.498981 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:18.498985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:18.499039 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:18.499120 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:18.499787 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.500455 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.501606 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.502012 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.502101 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:18.512132 (+ 10031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:51533: BlockingRecv error: recv got EOF from 127.3.186.194:51533 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 21:34:18 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:18 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:18.673058 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:18.664685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:54653)
0927 21:34:18.665254 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:18.665268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:18.665421 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:18.665753 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:18.665757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:18.666104 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:18.666256 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:18.666264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.667267 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.667270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:18.668168 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:18.668178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.668825 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.668829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:18.668943 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:18.669579 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:18.669601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:18.672829 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
W20260927 21:34:18.673184 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:18.664755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:54653 (local address 127.3.186.254:44171)
0927 21:34:18.665012 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:18.665020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:18.665394 (+ 374us) server_negotiation.cc:407] Connection header received
0927 21:34:18.665559 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:18.665565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:18.665637 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:18.665715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:18.666392 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.667159 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.668296 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:18.668679 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:18.668778 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:18.672966 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:54653: BlockingRecv error: recv got EOF from 127.3.186.195:54653 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 21:34:19 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:19 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:19.074342 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:19.065569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:56939)
0927 21:34:19.066099 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:19.066120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:19.066283 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:19.066652 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:19.066656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:19.067044 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:19.067187 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:19.067195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.068157 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.068163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:19.069067 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:19.069078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.069805 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.069810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:19.069992 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:19.070658 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:19.070685 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:19.074134 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":299,"threads_started":1}
W20260927 21:34:19.074536 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:19.065918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:56939 (local address 127.3.186.254:44171)
0927 21:34:19.066090 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:19.066095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:19.066213 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:34:19.066449 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:19.066452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:19.066507 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:19.066609 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:19.067327 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.068026 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.069204 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.069654 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.069780 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:19.074307 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:56939: BlockingRecv error: recv got EOF from 127.3.186.193:56939 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 21:34:19 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:19 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.194@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:19.521082 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:19.513200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.194:43313)
0927 21:34:19.513708 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:19.513725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:19.513830 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:19.514105 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:19.514108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:19.514487 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:19.514635 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:19.514644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.515543 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.515546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:19.516400 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:19.516409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.517089 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.517093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:19.517202 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:19.517819 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:19.517834 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:19.520873 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":269,"threads_started":1}
W20260927 21:34:19.521291 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:19.513304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:43313 (local address 127.3.186.254:44171)
0927 21:34:19.513461 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:19.513466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:19.513791 (+ 325us) server_negotiation.cc:407] Connection header received
0927 21:34:19.513926 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:19.513929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:19.513985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:19.514060 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:19.514745 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.515435 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.516517 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.516978 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.517115 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:19.521039 (+ 3924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.194:43313: BlockingRecv error: recv got EOF from 127.3.186.194:43313 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 21:34:19 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:19 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.195@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:19.681982 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:19.674147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.195:56085)
0927 21:34:19.674609 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:19.674624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:19.674731 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:19.674995 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:19.674999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:19.675347 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:19.675486 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:19.675494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.676381 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.676386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:19.677235 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:19.677244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.677930 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.677934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:19.678045 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:19.678546 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:19.678562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:19.681754 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":267,"threads_started":1}
W20260927 21:34:19.682163 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:19.674179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:56085 (local address 127.3.186.254:44171)
0927 21:34:19.674390 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:19.674399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:19.674699 (+ 300us) server_negotiation.cc:407] Connection header received
0927 21:34:19.674822 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:19.674825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:19.674886 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:19.674955 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:19.675601 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.676276 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.677373 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:19.677801 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:19.677889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:19.681949 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.195:56085: BlockingRecv error: recv got EOF from 127.3.186.195:56085 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 21:34:20 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:34:20 dist-test-slave-pqr8 krb5kdc[12056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.186.193@KRBTEST.COM for oryx/127.3.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:34:20.083523 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:20.075436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44171 (local address 127.3.186.193:38493)
0927 21:34:20.075889 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:20.075904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:20.076028 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:20.076327 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:20.076330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:20.076659 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:20.076822 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:20.076829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.077755 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.077759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:20.078746 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:20.078757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.079419 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.079424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:20.079590 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:20.080087 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:20.080102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:20.083179 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.186.254:44171: Server oryx/127.3.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":249,"threads_started":1}
W20260927 21:34:20.083627 13061 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:20.075464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:38493 (local address 127.3.186.254:44171)
0927 21:34:20.075818 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:20.075824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:20.076004 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:34:20.076119 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:20.076122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:20.076202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:20.076277 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:20.076932 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.077631 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.078879 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.079289 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.079411 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:20.083352 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.186.193:38493: BlockingRecv error: recv got EOF from 127.3.186.193:38493 (error 108)
Metrics: {"server-negotiator.queue_time_us":188}
I20260927 21:34:20.344863 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12643
I20260927 21:34:20.355850 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:20.345903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.3.186.193:39319)
0927 21:34:20.346453 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:20.346460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:20.346483 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:34:20.346535 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:20.346541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:20.346750 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:20.346899 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:20.348119 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.349090 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.350214 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.350710 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.350802 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:20.353098 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:20.353105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:20.353116 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:20.353150 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:20.355386 (+ 2236us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.186.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:34:20.355606 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34212: Request ticket server kudu/127.3.186.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
W20260927 21:34:20.355943 13105 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34212: Request ticket server kudu/127.3.186.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:34:20.355945 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12643: Not authorized: Client connection negotiation failed: client connection to 127.3.186.193:39319: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.186.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:34:20.356184 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12643
I20260927 21:34:20.372617 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12794
I20260927 21:34:20.384248 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:20.373480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.3.186.194:43201)
0927 21:34:20.374063 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:20.374071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:20.374098 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:34:20.374171 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:20.374178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:20.374452 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:20.374680 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:20.375766 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.377046 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.378272 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.378778 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.378881 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:20.381322 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:20.381329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:20.381341 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:20.381375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:20.383698 (+ 2323us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.186.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:34:20.383972 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50560: Request ticket server kudu/127.3.186.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":259,"threads_started":1}
W20260927 21:34:20.384379 13106 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50560: Request ticket server kudu/127.3.186.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:34:20.384452 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12794: Not authorized: Client connection negotiation failed: client connection to 127.3.186.194:43201: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.186.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:34:20.384590 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12794
I20260927 21:34:20.400256 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12927
W20260927 21:34:20.412971 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12927: Not authorized: Client connection negotiation failed: client connection to 127.3.186.195:33201: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.186.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:34:20.412983 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:20.401113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.3.186.195:33201)
0927 21:34:20.401686 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:20.401692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:20.401717 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:34:20.401771 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:20.401777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:20.401996 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:20.402146 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:20.403726 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.405369 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.407176 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.407665 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.407803 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:20.409949 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:20.409957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:20.409967 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:20.410001 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:20.412434 (+ 2433us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.186.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:34:20.412696 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36168: Request ticket server kudu/127.3.186.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
W20260927 21:34:20.413061 13107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36168: Request ticket server kudu/127.3.186.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:34:20.413129 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12927
I20260927 21:34:20.427914 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 12574
I20260927 21:34:20.428905 12636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:20.537318 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 12574
2026-09-27T21:34:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20661 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T21:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:34:20Z Disabled control of system clock
I20260927 21:34:20.607291 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41111
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:34:20.899978 13112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:20.900203 13112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:20.900272 13112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:20.906996 13112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:34:20.907078 13112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:20.907114 13112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:20.907167 13112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:34:20.907208 13112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:34:20.915385 13112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:41111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:20.916303 13112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:20.917845 13112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:20.928344 13117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:20.928269 13118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:20.928243 13120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:20.929165 13112 server_base.cc:1034] running on GCE node
I20260927 21:34:20.929909 13112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:20.931082 13112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:20.932276 13112 hybrid_clock.cc:648] HybridClock initialized: now 1790544860932257 us; error 39 us; skew 500 ppm
I20260927 21:34:20.932623 13112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:20.934716 13112 webserver.cc:460] Webserver started at http://127.3.186.254:39923/ using document root <none> and password file <none>
I20260927 21:34:20.935323 13112 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:20.935423 13112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:20.935732 13112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:20.938144 13112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "1a011336c15c42faa4aae55ae9d0d3f1"
format_stamp: "Formatted at 2026-09-27 21:34:20 on dist-test-slave-pqr8"
server_key: "ba1943a9d05092e60710741480f1e238"
server_key_iv: "09d8f14b67d9c7da656670eccc4a9237"
server_key_version: "encryptionkey@0"
I20260927 21:34:20.938889 13112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "1a011336c15c42faa4aae55ae9d0d3f1"
format_stamp: "Formatted at 2026-09-27 21:34:20 on dist-test-slave-pqr8"
server_key: "ba1943a9d05092e60710741480f1e238"
server_key_iv: "09d8f14b67d9c7da656670eccc4a9237"
server_key_version: "encryptionkey@0"
I20260927 21:34:20.943471 13112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 21:34:20.946391 13126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:20.947551 13112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:34:20.947722 13112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "1a011336c15c42faa4aae55ae9d0d3f1"
format_stamp: "Formatted at 2026-09-27 21:34:20 on dist-test-slave-pqr8"
server_key: "ba1943a9d05092e60710741480f1e238"
server_key_iv: "09d8f14b67d9c7da656670eccc4a9237"
server_key_version: "encryptionkey@0"
I20260927 21:34:20.947937 13112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:20.958626 13112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:20.959273 13112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:20.959547 13112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:20.976400 13112 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:41111
I20260927 21:34:20.976418 13177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:41111 every 8 connection(s)
I20260927 21:34:20.977582 13112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:34:20.980670 13178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:20.982982 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13112
I20260927 21:34:20.983144 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:34:20.983839 3819 external_mini_cluster.cc:1428] Setting key 90336983fa7ab8cc2d3a5e3eaadbc812
I20260927 21:34:20.991241 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1: Bootstrap starting.
I20260927 21:34:20.993971 13178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:20.994984 13178 log.cc:824] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:20.995906 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:20.985609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.3.186.254:41111)
0927 21:34:20.986397 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:20.986406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:20.986439 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:34:20.987927 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:20.987946 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:20.988318 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:20.988697 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:20.989222 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.990448 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.991732 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:20.992357 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:20.992555 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:20.993617 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:20.993624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:20.993635 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:20.994118 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:20.994216 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:20.994237 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:20.994382 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:20.994580 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 21:34:20.994830 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":280,"threads_started":1}
I20260927 21:34:20.997777 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1: No bootstrap required, opened a new log
I20260927 21:34:21.004624 13178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41111 } }
I20260927 21:34:21.004930 13178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:21.005028 13178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a011336c15c42faa4aae55ae9d0d3f1, State: Initialized, Role: FOLLOWER
I20260927 21:34:21.005481 13178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41111 } }
I20260927 21:34:21.005722 13178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:34:21.005856 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:34:21.006006 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:21.007856 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41111 } }
I20260927 21:34:21.008285 13178 leader_election.cc:302] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a011336c15c42faa4aae55ae9d0d3f1; no voters:
I20260927 21:34:21.008778 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:34:21.008948 13183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:34:21.010129 13183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 1a011336c15c42faa4aae55ae9d0d3f1, State: Running, Role: LEADER
I20260927 21:34:21.010571 13183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41111 } }
I20260927 21:34:21.010771 13178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:34:21.013398 13184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41111 } } }
I20260927 21:34:21.013329 13185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a011336c15c42faa4aae55ae9d0d3f1. Latest consensus state: current_term: 1 leader_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a011336c15c42faa4aae55ae9d0d3f1" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41111 } } }
I20260927 21:34:21.013762 13184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:21.013803 13185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:21.015741 13192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:34:21.020262 13192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:34:21.028368 13192 catalog_manager.cc:1347] Generated new cluster ID: 984e207f9e86418e817160c32c78c418
I20260927 21:34:21.028496 13192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:34:21.035059 13192 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:34:21.036027 13192 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:34:21.041680 13192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a011336c15c42faa4aae55ae9d0d3f1: Generated new TSK 0
I20260927 21:34:21.042312 13192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:34:21.056655 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41111
--builtin_ntp_servers=127.3.186.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:34:21.370373 13202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:21.370600 13202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:21.370693 13202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:21.377103 13202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:21.377194 13202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:21.377283 13202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:34:21.385349 13202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:21.386366 13202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:21.387938 13202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:21.400339 13207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:21.400261 13210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:21.400312 13208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:21.401782 13202 server_base.cc:1034] running on GCE node
I20260927 21:34:21.402320 13202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:21.403101 13202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:21.404313 13202 hybrid_clock.cc:648] HybridClock initialized: now 1790544861404253 us; error 56 us; skew 500 ppm
I20260927 21:34:21.404639 13202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:21.406703 13202 webserver.cc:460] Webserver started at http://127.3.186.193:34221/ using document root <none> and password file <none>
I20260927 21:34:21.407581 13202 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:21.407696 13202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:21.408080 13202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:21.411449 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "393f61e813c948079ab2979c69a432b1"
format_stamp: "Formatted at 2026-09-27 21:34:21 on dist-test-slave-pqr8"
server_key: "81ca4c8815beaf9866461d0aa3c13915"
server_key_iv: "f844d6735fa4c94bff8010e7cf5a4983"
server_key_version: "encryptionkey@0"
I20260927 21:34:21.412209 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "393f61e813c948079ab2979c69a432b1"
format_stamp: "Formatted at 2026-09-27 21:34:21 on dist-test-slave-pqr8"
server_key: "81ca4c8815beaf9866461d0aa3c13915"
server_key_iv: "f844d6735fa4c94bff8010e7cf5a4983"
server_key_version: "encryptionkey@0"
I20260927 21:34:21.417007 13202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 21:34:21.420063 13216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:21.421250 13202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:34:21.421452 13202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "393f61e813c948079ab2979c69a432b1"
format_stamp: "Formatted at 2026-09-27 21:34:21 on dist-test-slave-pqr8"
server_key: "81ca4c8815beaf9866461d0aa3c13915"
server_key_iv: "f844d6735fa4c94bff8010e7cf5a4983"
server_key_version: "encryptionkey@0"
I20260927 21:34:21.421720 13202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:21.442646 13202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:21.443336 13202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:21.443614 13202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:21.444379 13202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:21.445679 13202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:21.445792 13202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:21.445915 13202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:21.446007 13202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:21.476526 13202 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:39585
I20260927 21:34:21.476611 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:39585 every 8 connection(s)
I20260927 21:34:21.477658 13202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:34:21.478559 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13202
I20260927 21:34:21.478745 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:34:21.479485 3819 external_mini_cluster.cc:1428] Setting key abe066a23f9485b24c6c372089eb133f
I20260927 21:34:21.482183 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41111
--builtin_ntp_servers=127.3.186.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:34:21.489697 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:21.480169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:42573 (local address 127.3.186.254:41111)
0927 21:34:21.480340 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:21.480346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:21.481795 (+ 1449us) server_negotiation.cc:407] Connection header received
0927 21:34:21.483501 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:21.483508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:21.483587 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:21.483683 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:21.484602 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.485309 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.487216 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.487596 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.487720 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:21.489023 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:21.489027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:21.489031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:21.489046 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:21.489073 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:21.489083 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:21.489155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:21.489376 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:34:21.489441 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:34:21.490904 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:21.480643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41111 (local address 127.3.186.193:42573)
0927 21:34:21.481663 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:21.481697 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:21.483266 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:21.483871 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:21.483880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:21.483951 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:21.484457 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:21.484472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.485498 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.485502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:21.487080 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:21.487091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.487813 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.487820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:21.488037 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:21.488694 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:34:21.488700 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:21.488747 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:34:21.488837 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:21.489180 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:21.489192 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:21.489202 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:34:21.489363 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 21:34:21.489748 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":289,"threads_started":1}
I20260927 21:34:21.492252 13329 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41111
I20260927 21:34:21.492517 13329 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:21.493148 13329 heartbeater.cc:511] Master 127.3.186.254:41111 requested a full tablet report, sending...
I20260927 21:34:21.494840 13143 ts_manager.cc:194] Registered new tserver with Master: 393f61e813c948079ab2979c69a432b1 (127.3.186.193:39585)
I20260927 21:34:21.496168 13143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.193:42573
W20260927 21:34:21.783396 13332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:21.783615 13332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:21.783748 13332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:21.792694 13332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:21.792778 13332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:21.792951 13332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:34:21.801344 13332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:21.802480 13332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:21.804100 13332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:21.816843 13337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:21.816891 13340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:21.816922 13338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:21.817739 13332 server_base.cc:1034] running on GCE node
I20260927 21:34:21.818266 13332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:21.819072 13332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:21.820292 13332 hybrid_clock.cc:648] HybridClock initialized: now 1790544861820244 us; error 58 us; skew 500 ppm
I20260927 21:34:21.820595 13332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:21.822719 13332 webserver.cc:460] Webserver started at http://127.3.186.194:38413/ using document root <none> and password file <none>
I20260927 21:34:21.823372 13332 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:21.823494 13332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:21.823822 13332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:21.826555 13332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "d51bde94f74e464792004c45c037d4f5"
format_stamp: "Formatted at 2026-09-27 21:34:21 on dist-test-slave-pqr8"
server_key: "43ccdcd75ce20c1e33b1d7bb18e030ca"
server_key_iv: "becdd0ad85ee7c954ceed188f12fdc58"
server_key_version: "encryptionkey@0"
I20260927 21:34:21.827278 13332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "d51bde94f74e464792004c45c037d4f5"
format_stamp: "Formatted at 2026-09-27 21:34:21 on dist-test-slave-pqr8"
server_key: "43ccdcd75ce20c1e33b1d7bb18e030ca"
server_key_iv: "becdd0ad85ee7c954ceed188f12fdc58"
server_key_version: "encryptionkey@0"
I20260927 21:34:21.831840 13332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:34:21.834867 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:21.836000 13332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:34:21.836155 13332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "d51bde94f74e464792004c45c037d4f5"
format_stamp: "Formatted at 2026-09-27 21:34:21 on dist-test-slave-pqr8"
server_key: "43ccdcd75ce20c1e33b1d7bb18e030ca"
server_key_iv: "becdd0ad85ee7c954ceed188f12fdc58"
server_key_version: "encryptionkey@0"
I20260927 21:34:21.836349 13332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:21.855283 13332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:21.855947 13332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:21.856201 13332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:21.857005 13332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:21.858397 13332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:21.858513 13332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:21.858640 13332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:21.858726 13332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:21.889657 13332 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:43337
I20260927 21:34:21.889763 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:43337 every 8 connection(s)
I20260927 21:34:21.890733 13332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:34:21.892508 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13332
I20260927 21:34:21.892674 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:34:21.893172 3819 external_mini_cluster.cc:1428] Setting key 69e6f6fd76c82634199bfd9132ca1ae0
I20260927 21:34:21.896041 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41111
--builtin_ntp_servers=127.3.186.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:34:21.904450 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:21.893245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:55175 (local address 127.3.186.254:41111)
0927 21:34:21.893433 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:21.893440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:21.894841 (+ 1401us) server_negotiation.cc:407] Connection header received
0927 21:34:21.897270 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:21.897274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:21.897332 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:21.897415 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:21.898662 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.899306 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.901640 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.902015 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.902105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:21.903580 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:21.903589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:21.903595 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:21.903616 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:21.903656 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:21.903672 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:21.903759 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:21.903955 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:34:21.904048 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:34:21.905035 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:21.893714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41111 (local address 127.3.186.194:55175)
0927 21:34:21.894667 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:21.894716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:21.896963 (+ 2247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:21.897679 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:21.897691 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:21.897796 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:21.898505 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:21.898529 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.899459 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.899468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:21.901469 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:21.901485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:21.902219 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:21.902228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:21.902544 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:21.903250 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:34:21.903255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:21.903299 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:34:21.903377 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:21.903761 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:21.903768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:21.903773 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:34:21.903889 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:34:21.904129 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260927 21:34:21.906255 13459 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41111
I20260927 21:34:21.906519 13459 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:21.907151 13459 heartbeater.cc:511] Master 127.3.186.254:41111 requested a full tablet report, sending...
I20260927 21:34:21.908330 13143 ts_manager.cc:194] Registered new tserver with Master: d51bde94f74e464792004c45c037d4f5 (127.3.186.194:43337)
I20260927 21:34:21.909093 13143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.194:55175
W20260927 21:34:22.201720 13462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:22.201984 13462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:22.202128 13462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:22.208797 13462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:22.208907 13462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:22.209061 13462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:34:22.217761 13462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:22.218776 13462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:22.220371 13462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:22.232262 13467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:22.232295 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:22.232214 13468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:22.233183 13462 server_base.cc:1034] running on GCE node
I20260927 21:34:22.233801 13462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:22.234580 13462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:22.235795 13462 hybrid_clock.cc:648] HybridClock initialized: now 1790544862235761 us; error 47 us; skew 500 ppm
I20260927 21:34:22.236071 13462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:22.238041 13462 webserver.cc:460] Webserver started at http://127.3.186.195:40453/ using document root <none> and password file <none>
I20260927 21:34:22.238682 13462 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:22.238808 13462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:22.239140 13462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:22.241761 13462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "d0ae6ba7342340a49d4bef88fd5fc48b"
format_stamp: "Formatted at 2026-09-27 21:34:22 on dist-test-slave-pqr8"
server_key: "c7058e1c25647d48e03629c1b8d2f8ca"
server_key_iv: "1e5facb99294bc959079e8bc8919fcd6"
server_key_version: "encryptionkey@0"
I20260927 21:34:22.242498 13462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "d0ae6ba7342340a49d4bef88fd5fc48b"
format_stamp: "Formatted at 2026-09-27 21:34:22 on dist-test-slave-pqr8"
server_key: "c7058e1c25647d48e03629c1b8d2f8ca"
server_key_iv: "1e5facb99294bc959079e8bc8919fcd6"
server_key_version: "encryptionkey@0"
I20260927 21:34:22.246891 13462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:34:22.249847 13476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:22.251056 13462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:34:22.251219 13462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "d0ae6ba7342340a49d4bef88fd5fc48b"
format_stamp: "Formatted at 2026-09-27 21:34:22 on dist-test-slave-pqr8"
server_key: "c7058e1c25647d48e03629c1b8d2f8ca"
server_key_iv: "1e5facb99294bc959079e8bc8919fcd6"
server_key_version: "encryptionkey@0"
I20260927 21:34:22.251423 13462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:22.262686 13462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:22.263360 13462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:22.263641 13462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:22.264437 13462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:22.265686 13462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:22.265796 13462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:22.265910 13462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:22.266000 13462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:22.296134 13462 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:34893
I20260927 21:34:22.296201 13588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:34893 every 8 connection(s)
I20260927 21:34:22.297216 13462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:34:22.307340 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13462
I20260927 21:34:22.307489 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:34:22.307942 3819 external_mini_cluster.cc:1428] Setting key ed2fa4360f4e5762ca1c03eb92f8d2e0
I20260927 21:34:22.308156 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:22.299423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:55235 (local address 127.3.186.254:41111)
0927 21:34:22.299580 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:22.299584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:22.300756 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 21:34:22.302413 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:22.302418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:22.302512 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:22.302617 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:22.303581 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.304250 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.305708 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.306080 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.306207 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:22.307337 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:22.307347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:22.307350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:22.307365 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:22.307420 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:22.307435 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:22.307517 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:22.307759 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:34:22.307859 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:34:22.309201 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:22.299864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41111 (local address 127.3.186.195:55235)
0927 21:34:22.300621 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:22.300654 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:22.302162 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:22.302832 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:22.302839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:22.302942 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:22.303427 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:22.303442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.304398 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.304410 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:22.305552 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:22.305570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.306279 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.306287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:22.306494 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:22.307065 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:34:22.307068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:22.307104 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:34:22.307164 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:22.307516 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:22.307528 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:22.307538 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:34:22.307755 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 21:34:22.308107 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":228,"threads_started":1}
I20260927 21:34:22.310474 13589 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41111
I20260927 21:34:22.310766 13589 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:22.311385 13589 heartbeater.cc:511] Master 127.3.186.254:41111 requested a full tablet report, sending...
I20260927 21:34:22.312623 13143 ts_manager.cc:194] Registered new tserver with Master: d0ae6ba7342340a49d4bef88fd5fc48b (127.3.186.195:34893)
I20260927 21:34:22.313318 13143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.195:55235
I20260927 21:34:22.322646 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:34:22.328385 13181 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:22.326890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.3.186.254:41111)
0927 21:34:22.327011 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:22.327015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:22.327439 (+ 424us) server_negotiation.cc:407] Connection header received
0927 21:34:22.327637 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:22.327640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:22.327705 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:22.327798 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:22.328163 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52962: BlockingRecv error: recv got EOF from 127.0.0.1:52962 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:34:22.329221 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13202
I20260927 21:34:22.335777 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:22.330052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.3.186.193:39585)
0927 21:34:22.330511 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:22.330517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:22.330537 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:34:22.330590 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:22.330595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:22.330737 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:22.330838 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:22.331105 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.332046 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.333206 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.333718 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.333802 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:22.334619 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:22.334625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:22.334634 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:22.335105 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:22.335142 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:22.335159 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:22.335238 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:22.335339 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:34:22.335424 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":219,"threads_started":1}
I20260927 21:34:22.336695 13324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:22.414995 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13202
I20260927 21:34:22.430488 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13332
I20260927 21:34:22.437034 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:22.431189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.3.186.194:43337)
0927 21:34:22.431707 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:22.431714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:22.431739 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:34:22.431797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:22.431802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:22.431998 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:22.432121 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:22.432423 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.433297 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.434476 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.434935 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.435030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:22.435840 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:22.435859 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:22.435871 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:22.436377 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:22.436419 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:22.436435 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:22.436512 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:22.436596 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:34:22.436721 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":277,"threads_started":1}
I20260927 21:34:22.438177 13454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:22.514372 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13332
I20260927 21:34:22.529232 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13462
I20260927 21:34:22.536511 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:22.530034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.3.186.195:34893)
0927 21:34:22.530588 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:22.530595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:22.530620 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:34:22.530680 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:22.530686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:22.530876 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:22.531039 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:22.531489 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.532479 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.533671 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:22.534191 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:22.534315 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:22.535224 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:22.535232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:22.535243 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:22.535749 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:22.535787 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:22.535804 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:22.535928 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:22.536073 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 21:34:22.536195 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
I20260927 21:34:22.537701 13584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:22.615098 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13462
I20260927 21:34:22.630896 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13112
I20260927 21:34:22.631969 13173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:22.725804 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13112
2026-09-27T21:34:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2183 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T21:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:34:22Z Disabled control of system clock
I20260927 21:34:22.790251 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42599
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:35961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:34:23.088577 13608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:23.088812 13608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:23.088893 13608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:23.095475 13608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:34:23.095556 13608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:23.095589 13608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:23.095635 13608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:34:23.095674 13608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:34:23.103607 13608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:42599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:23.104691 13608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:23.106266 13608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:23.117142 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:23.117154 13614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:23.117190 13616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:23.117941 13608 server_base.cc:1034] running on GCE node
I20260927 21:34:23.118649 13608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:23.119796 13608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:23.120999 13608 hybrid_clock.cc:648] HybridClock initialized: now 1790544863120956 us; error 47 us; skew 500 ppm
I20260927 21:34:23.121321 13608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:23.123389 13608 webserver.cc:460] Webserver started at http://127.3.186.254:45033/ using document root <none> and password file <none>
I20260927 21:34:23.124037 13608 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:23.124158 13608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:23.124459 13608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:23.126993 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "3d82841a05fc486c84f825f4ee965cc9"
format_stamp: "Formatted at 2026-09-27 21:34:23 on dist-test-slave-pqr8"
server_key: "366e0935ad0108226848e98a777b7ae2"
server_key_iv: "c3d767b83fadcd492143f299d4578400"
server_key_version: "encryptionkey@0"
I20260927 21:34:23.127712 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "3d82841a05fc486c84f825f4ee965cc9"
format_stamp: "Formatted at 2026-09-27 21:34:23 on dist-test-slave-pqr8"
server_key: "366e0935ad0108226848e98a777b7ae2"
server_key_iv: "c3d767b83fadcd492143f299d4578400"
server_key_version: "encryptionkey@0"
I20260927 21:34:23.132246 13608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:34:23.135253 13622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:23.136325 13608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:34:23.136508 13608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "3d82841a05fc486c84f825f4ee965cc9"
format_stamp: "Formatted at 2026-09-27 21:34:23 on dist-test-slave-pqr8"
server_key: "366e0935ad0108226848e98a777b7ae2"
server_key_iv: "c3d767b83fadcd492143f299d4578400"
server_key_version: "encryptionkey@0"
I20260927 21:34:23.136698 13608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:23.141757 13608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:23.142431 13608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:23.142731 13608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:23.159340 13608 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:42599
I20260927 21:34:23.159380 13673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:42599 every 8 connection(s)
I20260927 21:34:23.160490 13608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:34:23.163730 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:23.165179 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13608
I20260927 21:34:23.165376 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:34:23.166196 3819 external_mini_cluster.cc:1428] Setting key 1c44231f872b22084262c3a05d5150c8
I20260927 21:34:23.173456 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:23.167989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.3.186.254:42599)
0927 21:34:23.168909 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:23.168918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:23.168962 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:34:23.170355 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:23.170379 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:23.170729 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:23.171033 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:23.172009 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:23.172015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:23.172026 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:23.172043 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:23.172092 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:23.172105 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:23.172257 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:23.172396 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:34:23.172647 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":384,"threads_started":1}
I20260927 21:34:23.176374 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9: Bootstrap starting.
I20260927 21:34:23.180130 13674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:23.181335 13674 log.cc:824] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:23.183722 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9: No bootstrap required, opened a new log
I20260927 21:34:23.189956 13674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d82841a05fc486c84f825f4ee965cc9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42599 } }
I20260927 21:34:23.190218 13674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:23.190282 13674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d82841a05fc486c84f825f4ee965cc9, State: Initialized, Role: FOLLOWER
I20260927 21:34:23.190694 13674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d82841a05fc486c84f825f4ee965cc9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42599 } }
I20260927 21:34:23.190866 13674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:34:23.190951 13674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:34:23.191046 13674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:23.193099 13674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d82841a05fc486c84f825f4ee965cc9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42599 } }
I20260927 21:34:23.193475 13674 leader_election.cc:302] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d82841a05fc486c84f825f4ee965cc9; no voters:
I20260927 21:34:23.194065 13674 leader_election.cc:288] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:34:23.194161 13679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:34:23.194856 13679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d82841a05fc486c84f825f4ee965cc9, State: Running, Role: LEADER
I20260927 21:34:23.195356 13679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d82841a05fc486c84f825f4ee965cc9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42599 } }
I20260927 21:34:23.195874 13674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:34:23.198050 13680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d82841a05fc486c84f825f4ee965cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d82841a05fc486c84f825f4ee965cc9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42599 } } }
I20260927 21:34:23.198503 13680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:23.198258 13681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d82841a05fc486c84f825f4ee965cc9. Latest consensus state: current_term: 1 leader_uuid: "3d82841a05fc486c84f825f4ee965cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d82841a05fc486c84f825f4ee965cc9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42599 } } }
I20260927 21:34:23.198653 13681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:23.202909 13685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:34:23.207721 13685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:34:23.215189 13685 catalog_manager.cc:1347] Generated new cluster ID: c9a003aa63d24ac3a7c478b058b35f10
I20260927 21:34:23.215282 13685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:34:23.234092 13685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:34:23.234493 13685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:34:23.244174 13685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d82841a05fc486c84f825f4ee965cc9: Generated new TSK 0
I20260927 21:34:23.244846 13685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:34:23.254493 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42599
--builtin_ntp_servers=127.3.186.212:35961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:34:23.581730 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:23.581974 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:23.582056 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:23.588851 13698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:23.588940 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:23.589038 13698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:34:23.597644 13698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:23.598644 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:23.600219 13698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:23.612392 13703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:23.612485 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:23.612380 13704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:23.613250 13698 server_base.cc:1034] running on GCE node
I20260927 21:34:23.613829 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:23.614652 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:23.615856 13698 hybrid_clock.cc:648] HybridClock initialized: now 1790544863615811 us; error 39 us; skew 500 ppm
I20260927 21:34:23.616174 13698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:23.624670 13698 webserver.cc:460] Webserver started at http://127.3.186.193:32883/ using document root <none> and password file <none>
I20260927 21:34:23.625342 13698 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:23.625454 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:23.625782 13698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:23.628441 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "8870d1b7ec844cd397d139d345e31e5c"
format_stamp: "Formatted at 2026-09-27 21:34:23 on dist-test-slave-pqr8"
server_key: "1307bfbfb0d4e5f5092bd23e614c4a2b"
server_key_iv: "c6b0d3839e475ae68fcd5d0089e76713"
server_key_version: "encryptionkey@0"
I20260927 21:34:23.629177 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "8870d1b7ec844cd397d139d345e31e5c"
format_stamp: "Formatted at 2026-09-27 21:34:23 on dist-test-slave-pqr8"
server_key: "1307bfbfb0d4e5f5092bd23e614c4a2b"
server_key_iv: "c6b0d3839e475ae68fcd5d0089e76713"
server_key_version: "encryptionkey@0"
I20260927 21:34:23.633808 13698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:34:23.636879 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:23.638139 13698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:34:23.638334 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "8870d1b7ec844cd397d139d345e31e5c"
format_stamp: "Formatted at 2026-09-27 21:34:23 on dist-test-slave-pqr8"
server_key: "1307bfbfb0d4e5f5092bd23e614c4a2b"
server_key_iv: "c6b0d3839e475ae68fcd5d0089e76713"
server_key_version: "encryptionkey@0"
I20260927 21:34:23.638541 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:23.643836 13698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:23.644520 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:23.644783 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:23.645715 13698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:23.646965 13698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:23.647082 13698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:23.647212 13698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:23.647296 13698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:23.678776 13698 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:40779
I20260927 21:34:23.678848 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:40779 every 8 connection(s)
I20260927 21:34:23.679970 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:34:23.686331 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13698
I20260927 21:34:23.686538 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:34:23.687143 3819 external_mini_cluster.cc:1428] Setting key 392d95959afecfdf2301f8144b666001
I20260927 21:34:23.687706 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:23.682386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:52627 (local address 127.3.186.254:42599)
0927 21:34:23.682896 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:23.682900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:23.683867 (+ 967us) server_negotiation.cc:407] Connection header received
0927 21:34:23.685468 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:23.685474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:23.685568 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:23.685687 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:23.686958 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:23.686964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:23.686968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:23.686985 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:23.687031 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:23.687043 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:23.687191 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:23.687383 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 21:34:23.687453 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":337,"threads_started":1}
I20260927 21:34:23.688640 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:23.682952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42599 (local address 127.3.186.193:52627)
0927 21:34:23.683729 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:23.683766 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:23.685210 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:23.685906 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:23.685914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:23.686028 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:23.686690 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:34:23.686692 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:23.686726 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:34:23.686801 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:23.687165 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:23.687175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:23.687182 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:34:23.687459 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 21:34:23.687672 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":234,"threads_started":1}
I20260927 21:34:23.689788 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42599
--builtin_ntp_servers=127.3.186.212:35961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 21:34:23.690038 13825 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42599
I20260927 21:34:23.690281 13825 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:23.690774 13825 heartbeater.cc:511] Master 127.3.186.254:42599 requested a full tablet report, sending...
I20260927 21:34:23.692576 13639 ts_manager.cc:194] Registered new tserver with Master: 8870d1b7ec844cd397d139d345e31e5c (127.3.186.193:40779)
W20260927 21:34:23.987216 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:23.987437 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:23.987514 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:23.994009 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:23.994081 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:23.994189 13829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:34:24.002643 13829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:24.003656 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:24.005088 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:24.016501 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:24.016597 13835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:24.016564 13834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:24.017729 13829 server_base.cc:1034] running on GCE node
I20260927 21:34:24.018258 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:24.019132 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:24.020342 13829 hybrid_clock.cc:648] HybridClock initialized: now 1790544864020312 us; error 43 us; skew 500 ppm
I20260927 21:34:24.020612 13829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:24.022672 13829 webserver.cc:460] Webserver started at http://127.3.186.194:42967/ using document root <none> and password file <none>
I20260927 21:34:24.023255 13829 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:24.023382 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:24.023692 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:24.026122 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "26284d34f200496784d55f73f2df0ad6"
format_stamp: "Formatted at 2026-09-27 21:34:24 on dist-test-slave-pqr8"
server_key: "d0aea2faee3c9544f7e60dee8723b641"
server_key_iv: "b0087773fca99a0ea376d40f39e617da"
server_key_version: "encryptionkey@0"
I20260927 21:34:24.026821 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "26284d34f200496784d55f73f2df0ad6"
format_stamp: "Formatted at 2026-09-27 21:34:24 on dist-test-slave-pqr8"
server_key: "d0aea2faee3c9544f7e60dee8723b641"
server_key_iv: "b0087773fca99a0ea376d40f39e617da"
server_key_version: "encryptionkey@0"
I20260927 21:34:24.031281 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:34:24.034466 13843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:24.035601 13829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:34:24.035768 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "26284d34f200496784d55f73f2df0ad6"
format_stamp: "Formatted at 2026-09-27 21:34:24 on dist-test-slave-pqr8"
server_key: "d0aea2faee3c9544f7e60dee8723b641"
server_key_iv: "b0087773fca99a0ea376d40f39e617da"
server_key_version: "encryptionkey@0"
I20260927 21:34:24.035982 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:24.041139 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:24.041908 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:24.042173 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:24.042979 13829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:24.044212 13829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:24.044333 13829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:24.044466 13829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:24.044555 13829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:24.075487 13829 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:45715
I20260927 21:34:24.075560 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:45715 every 8 connection(s)
I20260927 21:34:24.076649 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:34:24.078513 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13829
I20260927 21:34:24.078677 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:34:24.079154 3819 external_mini_cluster.cc:1428] Setting key fa8488d0c416bf6eddcc27c4ad099c6b
I20260927 21:34:24.081866 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42599
--builtin_ntp_servers=127.3.186.212:35961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 21:34:24.084470 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.079034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:33063 (local address 127.3.186.254:42599)
0927 21:34:24.079238 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:24.079246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:24.080598 (+ 1352us) server_negotiation.cc:407] Connection header received
0927 21:34:24.082370 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:24.082375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:24.082455 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:24.082568 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:24.083774 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:24.083779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:24.083784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:24.083801 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:24.083839 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:24.083850 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:24.083967 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:24.084153 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:34:24.084213 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:34:24.085258 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.079549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42599 (local address 127.3.186.194:33063)
0927 21:34:24.080469 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:24.080498 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:24.082123 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:24.082776 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:24.082784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:24.082865 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:24.083513 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:34:24.083516 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:24.083551 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:34:24.083623 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:24.083944 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:24.083954 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:24.083961 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:34:24.084227 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 21:34:24.084447 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":305,"threads_started":1}
I20260927 21:34:24.086551 13956 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42599
I20260927 21:34:24.086810 13956 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:24.087311 13956 heartbeater.cc:511] Master 127.3.186.254:42599 requested a full tablet report, sending...
I20260927 21:34:24.088721 13639 ts_manager.cc:194] Registered new tserver with Master: 26284d34f200496784d55f73f2df0ad6 (127.3.186.194:45715)
W20260927 21:34:24.398234 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:24.398461 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:24.398612 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:24.405210 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:24.405290 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:24.405462 13959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:34:24.414077 13959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:35961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:24.415117 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:24.416692 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:24.429765 13967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:24.429798 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:24.430022 13965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:24.430609 13959 server_base.cc:1034] running on GCE node
I20260927 21:34:24.431077 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:24.431890 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:24.433113 13959 hybrid_clock.cc:648] HybridClock initialized: now 1790544864433029 us; error 73 us; skew 500 ppm
I20260927 21:34:24.433403 13959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:34:24.435365 13959 webserver.cc:460] Webserver started at http://127.3.186.195:35757/ using document root <none> and password file <none>
I20260927 21:34:24.435976 13959 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:24.436095 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:24.436395 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:24.439067 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "562bad8a94ec4ec9b325ae0e92eea115"
format_stamp: "Formatted at 2026-09-27 21:34:24 on dist-test-slave-pqr8"
server_key: "a3bd830139bb7b6344df6a0d91bcbbc4"
server_key_iv: "c517ca6229c8605e56c03f5f8af126d9"
server_key_version: "encryptionkey@0"
I20260927 21:34:24.439859 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "562bad8a94ec4ec9b325ae0e92eea115"
format_stamp: "Formatted at 2026-09-27 21:34:24 on dist-test-slave-pqr8"
server_key: "a3bd830139bb7b6344df6a0d91bcbbc4"
server_key_iv: "c517ca6229c8605e56c03f5f8af126d9"
server_key_version: "encryptionkey@0"
I20260927 21:34:24.444499 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 21:34:24.447459 13973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:24.448531 13959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:34:24.448683 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "562bad8a94ec4ec9b325ae0e92eea115"
format_stamp: "Formatted at 2026-09-27 21:34:24 on dist-test-slave-pqr8"
server_key: "a3bd830139bb7b6344df6a0d91bcbbc4"
server_key_iv: "c517ca6229c8605e56c03f5f8af126d9"
server_key_version: "encryptionkey@0"
I20260927 21:34:24.448869 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:24.453758 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:24.454433 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:24.454702 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:24.455524 13959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:24.456722 13959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:24.456835 13959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:24.456969 13959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:24.457052 13959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:24.486935 13959 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:46297
I20260927 21:34:24.487010 14085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:46297 every 8 connection(s)
I20260927 21:34:24.488080 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:34:24.490975 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 13959
I20260927 21:34:24.491144 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:34:24.491652 3819 external_mini_cluster.cc:1428] Setting key 8997a92b139151496ef54027bb9691ee
I20260927 21:34:24.495739 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.490308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:45617 (local address 127.3.186.254:42599)
0927 21:34:24.490465 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:24.490469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:24.491815 (+ 1346us) server_negotiation.cc:407] Connection header received
0927 21:34:24.493846 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:24.493850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:24.493917 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:24.493993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:24.495189 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:24.495192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:24.495195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:24.495209 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:24.495235 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:24.495246 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:24.495334 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:24.495480 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:34:24.495528 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:34:24.496573 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.490712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42599 (local address 127.3.186.195:45617)
0927 21:34:24.491667 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:24.491712 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:24.493578 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:24.494232 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:24.494243 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:24.494355 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:24.494944 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:34:24.494951 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:24.494982 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:34:24.495052 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:24.495325 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:24.495332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:24.495336 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:34:24.495524 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 21:34:24.495771 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":282,"threads_started":1}
I20260927 21:34:24.497843 14086 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42599
I20260927 21:34:24.498075 14086 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:24.498512 14086 heartbeater.cc:511] Master 127.3.186.254:42599 requested a full tablet report, sending...
I20260927 21:34:24.499630 13639 ts_manager.cc:194] Registered new tserver with Master: 562bad8a94ec4ec9b325ae0e92eea115 (127.3.186.195:46297)
I20260927 21:34:24.506470 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:34:24.511880 13827 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:34:24.510545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.3.186.254:42599)
0927 21:34:24.510676 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:24.510679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:24.511148 (+ 469us) server_negotiation.cc:407] Connection header received
0927 21:34:24.511251 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:24.511254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:24.511291 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:24.511360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:24.511682 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57386: BlockingRecv error: recv got EOF from 127.0.0.1:57386 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:34:24.512504 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13698
I20260927 21:34:24.516434 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.513568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.3.186.193:40779)
0927 21:34:24.514064 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:24.514070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:24.514181 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:34:24.514336 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:24.514341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:24.514494 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:24.514601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:24.515727 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:24.515734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:24.515746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:24.515765 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:24.515833 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:24.515851 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:24.515959 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:24.516059 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:34:24.516148 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260927 21:34:24.517428 13820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:24.594043 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13698
I20260927 21:34:24.610975 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13829
I20260927 21:34:24.614377 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.611781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.3.186.194:45715)
0927 21:34:24.612583 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:24.612590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:24.612616 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:34:24.612687 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:24.612693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:24.612889 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:24.613005 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:24.613829 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:24.613835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:24.613845 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:24.613856 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:24.613896 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:24.613911 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:24.614005 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:24.614083 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:34:24.614155 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":518,"threads_started":1}
I20260927 21:34:24.615490 13951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:24.692246 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13829
I20260927 21:34:24.708618 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13959
I20260927 21:34:24.712440 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:24.709559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.3.186.195:46297)
0927 21:34:24.710142 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:24.710148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:24.710176 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:34:24.710244 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:24.710249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:24.710461 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:24.710629 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:24.711722 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:24.711728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:24.711738 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:34:24.711753 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:24.711788 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:34:24.711800 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:24.711982 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:24.712057 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:34:24.712153 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":284,"threads_started":1}
I20260927 21:34:24.713645 14081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:24.791692 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13959
I20260927 21:34:24.807370 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 13608
I20260927 21:34:24.808450 13669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:24.901376 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 13608
2026-09-27T21:34:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2175 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:34:24 dist-test-slave-pqr8 krb5kdc[14104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:34:24 dist-test-slave-pqr8 krb5kdc[14104](info): set up 2 sockets
Sep 27 21:34:24 dist-test-slave-pqr8 krb5kdc[14104](info): commencing operation
krb5kdc: starting...
W20260927 21:34:25.954216 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:34:26 dist-test-slave-pqr8 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:34:26.149467 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41103
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:41103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:26.457041 14117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:26.457286 14117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:26.457366 14117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:26.464129 14117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:34:26.464218 14117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:26.464262 14117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:26.464315 14117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:34:26.464357 14117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:34:26.473244 14117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:41103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:26.474305 14117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:26.475399 14117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:26.485930 14122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:26.485981 14125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:26.485998 14123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:26.487033 14117 server_base.cc:1034] running on GCE node
I20260927 21:34:26.487728 14117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:26.489032 14117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:26.490253 14117 hybrid_clock.cc:648] HybridClock initialized: now 1790544866490230 us; error 46 us; skew 500 ppm
I20260927 21:34:26.490552 14117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:26 dist-test-slave-pqr8 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544866, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:26.493372 14117 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:34:26.494699 14117 webserver.cc:460] Webserver started at http://127.3.186.254:39051/ using document root <none> and password file <none>
I20260927 21:34:26.495290 14117 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:26.495424 14117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:26.495695 14117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:26.498358 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "261554f320f94701893553d0aa5a7b2d"
format_stamp: "Formatted at 2026-09-27 21:34:26 on dist-test-slave-pqr8"
server_key: "5db5f3bceeb3bf05d37fdf6772924596"
server_key_iv: "456921d229e97eb6c4bdce3d3505cbfb"
server_key_version: "encryptionkey@0"
I20260927 21:34:26.499035 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "261554f320f94701893553d0aa5a7b2d"
format_stamp: "Formatted at 2026-09-27 21:34:26 on dist-test-slave-pqr8"
server_key: "5db5f3bceeb3bf05d37fdf6772924596"
server_key_iv: "456921d229e97eb6c4bdce3d3505cbfb"
server_key_version: "encryptionkey@0"
I20260927 21:34:26.503707 14117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:34:26.506843 14132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:26.508038 14117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:34:26.508208 14117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "261554f320f94701893553d0aa5a7b2d"
format_stamp: "Formatted at 2026-09-27 21:34:26 on dist-test-slave-pqr8"
server_key: "5db5f3bceeb3bf05d37fdf6772924596"
server_key_iv: "456921d229e97eb6c4bdce3d3505cbfb"
server_key_version: "encryptionkey@0"
I20260927 21:34:26.508383 14117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:26.556740 14117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:26.560218 14117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:26.560492 14117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:26.577720 14117 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:41103
I20260927 21:34:26.577790 14183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:41103 every 8 connection(s)
I20260927 21:34:26.578905 14117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:34:26.582532 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:26.585836 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 14117
I20260927 21:34:26.586113 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:34:26.587001 3819 external_mini_cluster.cc:1428] Setting key 779fd996c499952ff955f54d58b86fbc
I20260927 21:34:26.595325 14184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d: Bootstrap starting.
I20260927 21:34:26.598798 14184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:26.600008 14184 log.cc:824] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:34:26 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:26.603047 14184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d: No bootstrap required, opened a new log
I20260927 21:34:26.610093 14187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:26.589236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.3.186.254:41103)
0927 21:34:26.589982 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:26.589992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:26.590027 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:34:26.591538 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:26.591557 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:26.591898 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:26.592206 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:26.593920 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:26.595093 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:26.596781 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:26.597378 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:26.597500 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:26.602144 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:26.602157 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:26.602174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:26.602214 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:26.606133 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:26.606704 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:26.606713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:26.606724 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:26.606904 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:26.607256 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:26.607260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:26.607264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:26.607845 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:26.608162 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:26.608426 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:34:26.608826 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":378,"threads_started":1}
I20260927 21:34:26.613348 14184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261554f320f94701893553d0aa5a7b2d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41103 } }
I20260927 21:34:26.613703 14184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:26.613801 14184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261554f320f94701893553d0aa5a7b2d, State: Initialized, Role: FOLLOWER
I20260927 21:34:26.614322 14184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261554f320f94701893553d0aa5a7b2d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41103 } }
I20260927 21:34:26.614513 14184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:34:26.614603 14184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:34:26.614715 14184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:26.616495 14184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261554f320f94701893553d0aa5a7b2d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41103 } }
I20260927 21:34:26.616853 14184 leader_election.cc:302] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261554f320f94701893553d0aa5a7b2d; no voters:
I20260927 21:34:26.617414 14184 leader_election.cc:288] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:34:26.617542 14189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:34:26.618526 14189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [term 1 LEADER]: Becoming Leader. State: Replica: 261554f320f94701893553d0aa5a7b2d, State: Running, Role: LEADER
I20260927 21:34:26.619057 14189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261554f320f94701893553d0aa5a7b2d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41103 } }
I20260927 21:34:26.619647 14184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:34:26.621824 14190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261554f320f94701893553d0aa5a7b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261554f320f94701893553d0aa5a7b2d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41103 } } }
I20260927 21:34:26.622299 14190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:26.621855 14191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261554f320f94701893553d0aa5a7b2d. Latest consensus state: current_term: 1 leader_uuid: "261554f320f94701893553d0aa5a7b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261554f320f94701893553d0aa5a7b2d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41103 } } }
I20260927 21:34:26.622866 14191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d [sys.catalog]: This master's current role is: LEADER
I20260927 21:34:26.624424 14196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:34:26.629199 14196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:34:26.636575 14196 catalog_manager.cc:1347] Generated new cluster ID: da26281df57b4978ad939511435263d4
I20260927 21:34:26.636669 14196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:34:26.653501 14196 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:34:26.655179 14196 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:34:26.667392 14196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d: Generated new TSK 0
I20260927 21:34:26.668119 14196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:34:26.725072 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41103
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:27.026253 14210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:27.026479 14210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:27.026638 14210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:27.033267 14210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:27.033375 14210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:27.033625 14210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:34:27.042588 14210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:27.043642 14210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:27.044797 14210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:27.057145 14215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:27.057092 14218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:27.057201 14216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:27.057976 14210 server_base.cc:1034] running on GCE node
I20260927 21:34:27.058497 14210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:27.059299 14210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:27.060523 14210 hybrid_clock.cc:648] HybridClock initialized: now 1790544867060499 us; error 53 us; skew 500 ppm
I20260927 21:34:27.060827 14210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:27 dist-test-slave-pqr8 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544867, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:27.063776 14210 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:34:27.064930 14210 webserver.cc:460] Webserver started at http://127.3.186.193:40281/ using document root <none> and password file <none>
I20260927 21:34:27.065485 14210 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:27.065683 14210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:27.065999 14210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:27.068674 14210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "cd1c80cd3308479da4b21c601f291e41"
format_stamp: "Formatted at 2026-09-27 21:34:27 on dist-test-slave-pqr8"
server_key: "490bf76a578c5fc02385e0ed74e67c45"
server_key_iv: "7f8cebe0c835b51430b475e88cb93a4e"
server_key_version: "encryptionkey@0"
I20260927 21:34:27.069429 14210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "cd1c80cd3308479da4b21c601f291e41"
format_stamp: "Formatted at 2026-09-27 21:34:27 on dist-test-slave-pqr8"
server_key: "490bf76a578c5fc02385e0ed74e67c45"
server_key_iv: "7f8cebe0c835b51430b475e88cb93a4e"
server_key_version: "encryptionkey@0"
I20260927 21:34:27.074025 14210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:34:27.077069 14225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:27.078318 14210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:34:27.078472 14210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "cd1c80cd3308479da4b21c601f291e41"
format_stamp: "Formatted at 2026-09-27 21:34:27 on dist-test-slave-pqr8"
server_key: "490bf76a578c5fc02385e0ed74e67c45"
server_key_iv: "7f8cebe0c835b51430b475e88cb93a4e"
server_key_version: "encryptionkey@0"
I20260927 21:34:27.078652 14210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:27.105172 14210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:27.108752 14210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:27.109025 14210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:27.109923 14210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:27.111065 14210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:27.111162 14210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:27.111295 14210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:27.111363 14210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:27.143538 14210 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:34881
I20260927 21:34:27.143611 14337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:34881 every 8 connection(s)
I20260927 21:34:27.144656 14210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:34:27.152848 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 14210
I20260927 21:34:27.153057 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:34:27.153683 3819 external_mini_cluster.cc:1428] Setting key 6321dd407da675ea09afcac75ecc566f
Sep 27 21:34:27 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544867, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:27.163566 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:27.146905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:48449 (local address 127.3.186.254:41103)
0927 21:34:27.149314 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:27.149318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:27.149334 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:34:27.150562 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:27.150566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:27.150626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:27.150721 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:27.151985 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.152699 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.154421 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.154959 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.155078 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:27.158306 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:27.158311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:27.158314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:27.158341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:27.160720 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:27.161387 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:27.161392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:27.161395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:27.161561 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:27.161999 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:27.162004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:27.162007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:27.162262 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:27.162413 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:27.163020 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 21:34:27.163226 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2306,"thread_start_us":2227,"threads_started":1}
I20260927 21:34:27.164378 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:27.147548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41103 (local address 127.3.186.193:48449)
0927 21:34:27.148562 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:27.148636 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:27.150292 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:27.150894 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:27.150901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:27.151341 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:27.151846 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:27.151861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.152873 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.152880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:27.154269 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:27.154280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.155188 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.155198 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:27.155437 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:27.156028 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:27.156048 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:27.158089 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:27.160856 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:27.160864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:27.160877 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:27.161261 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:27.161688 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:27.161694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:27.161698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:27.161875 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:27.162397 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:27.162404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:27.162677 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 21:34:27.162921 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 21:34:27.163357 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":424,"threads_started":1}
I20260927 21:34:27.165864 14338 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41103
I20260927 21:34:27.166136 14338 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:27.167095 14338 heartbeater.cc:511] Master 127.3.186.254:41103 requested a full tablet report, sending...
I20260927 21:34:27.169258 14149 ts_manager.cc:194] Registered new tserver with Master: cd1c80cd3308479da4b21c601f291e41 (127.3.186.193:34881)
I20260927 21:34:27.171252 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:48449
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:34:27.200055 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41103
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:27.498384 14344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:27.498605 14344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:27.498683 14344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:27.505287 14344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:27.505366 14344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:27.505460 14344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:34:27.514480 14344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:27.515494 14344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:27.516587 14344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:27.528698 14352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:27.528784 14350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:27.528818 14349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:27.529660 14344 server_base.cc:1034] running on GCE node
I20260927 21:34:27.530210 14344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:27.531021 14344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:27.532238 14344 hybrid_clock.cc:648] HybridClock initialized: now 1790544867532197 us; error 57 us; skew 500 ppm
I20260927 21:34:27.532516 14344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:27 dist-test-slave-pqr8 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544867, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:27.535427 14344 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:34:27.536674 14344 webserver.cc:460] Webserver started at http://127.3.186.194:33863/ using document root <none> and password file <none>
I20260927 21:34:27.537267 14344 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:27.537401 14344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:27.537715 14344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:27.540211 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "c7fba8721f084feb9704586920504955"
format_stamp: "Formatted at 2026-09-27 21:34:27 on dist-test-slave-pqr8"
server_key: "9d33a4ae17aa2dccde0408180fc50d71"
server_key_iv: "31979aad0f3cc3df9f1265dd2290c58c"
server_key_version: "encryptionkey@0"
I20260927 21:34:27.540901 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "c7fba8721f084feb9704586920504955"
format_stamp: "Formatted at 2026-09-27 21:34:27 on dist-test-slave-pqr8"
server_key: "9d33a4ae17aa2dccde0408180fc50d71"
server_key_iv: "31979aad0f3cc3df9f1265dd2290c58c"
server_key_version: "encryptionkey@0"
I20260927 21:34:27.545260 14344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 21:34:27.548180 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:27.549366 14344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:34:27.549556 14344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "c7fba8721f084feb9704586920504955"
format_stamp: "Formatted at 2026-09-27 21:34:27 on dist-test-slave-pqr8"
server_key: "9d33a4ae17aa2dccde0408180fc50d71"
server_key_iv: "31979aad0f3cc3df9f1265dd2290c58c"
server_key_version: "encryptionkey@0"
I20260927 21:34:27.549754 14344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:27.596154 14344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:27.600181 14344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:27.600396 14344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:27.601147 14344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:27.602411 14344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:27.602476 14344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:27.602553 14344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:27.602595 14344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:27.634742 14344 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:36595
I20260927 21:34:27.634935 14471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:36595 every 8 connection(s)
I20260927 21:34:27.635941 14344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:34:27.636878 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 14344
I20260927 21:34:27.637068 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:34:27.637598 3819 external_mini_cluster.cc:1428] Setting key b7198e843d8007e6f42e223225ef275b
Sep 27 21:34:27 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544867, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:27.657415 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:27.638478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:47989 (local address 127.3.186.254:41103)
0927 21:34:27.638728 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:27.638734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:27.639898 (+ 1164us) server_negotiation.cc:407] Connection header received
0927 21:34:27.642669 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:27.642673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:27.642736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:27.642833 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:27.644742 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.645438 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.647820 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.648234 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.648363 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:27.652030 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:27.652036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:27.652044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:27.652085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:27.654655 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:27.655313 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:27.655319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:27.655323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:27.655449 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:27.655903 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:27.655908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:27.655911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:27.656141 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:27.656295 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:27.656881 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 21:34:27.657090 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:34:27.658218 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:27.639013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41103 (local address 127.3.186.194:47989)
0927 21:34:27.639729 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:27.639779 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:27.642393 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:27.643071 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:27.643083 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:27.643792 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:27.644546 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:27.644573 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.645622 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.645630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:27.647669 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:27.647686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:27.648451 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:27.648458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:27.648670 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:27.649315 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:27.649345 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:27.651799 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:27.654808 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:27.654816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:27.654829 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:27.655199 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:27.655580 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:27.655586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:27.655589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:27.655773 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:27.656259 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:27.656266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:27.656554 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 21:34:27.656784 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 21:34:27.657207 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":285,"threads_started":1}
I20260927 21:34:27.659523 14472 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41103
I20260927 21:34:27.659801 14472 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:27.660475 14472 heartbeater.cc:511] Master 127.3.186.254:41103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
I20260927 21:34:27.661878 14149 ts_manager.cc:194] Registered new tserver with Master: c7fba8721f084feb9704586920504955 (127.3.186.194:36595)
I20260927 21:34:27.662638 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:47989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:34:27.684731 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41103
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:34:27.987020 14477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:34:27.987308 14477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:34:27.987462 14477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:34:27.994207 14477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:34:27.994333 14477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:34:27.994500 14477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:34:28.003342 14477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:41103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:34:28.004416 14477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:34:28.005574 14477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:34:28.017547 14485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:28.017590 14482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:34:28.017613 14483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:34:28.018327 14477 server_base.cc:1034] running on GCE node
I20260927 21:34:28.018858 14477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:34:28.019731 14477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:34:28.020912 14477 hybrid_clock.cc:648] HybridClock initialized: now 1790544868020854 us; error 58 us; skew 500 ppm
I20260927 21:34:28.021155 14477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544868, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:34:28.024011 14477 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:34:28.025305 14477 webserver.cc:460] Webserver started at http://127.3.186.195:41103/ using document root <none> and password file <none>
I20260927 21:34:28.026002 14477 fs_manager.cc:359] Metadata directory not provided
I20260927 21:34:28.026136 14477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:34:28.026418 14477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:34:28.028891 14477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "b632fe78821b4594a2776f0672e7a5a1"
format_stamp: "Formatted at 2026-09-27 21:34:28 on dist-test-slave-pqr8"
server_key: "b9bca34410f3326da1d38e2981b28081"
server_key_iv: "7b4c5299d936b27d48f2fa5c32d89975"
server_key_version: "encryptionkey@0"
I20260927 21:34:28.029671 14477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "b632fe78821b4594a2776f0672e7a5a1"
format_stamp: "Formatted at 2026-09-27 21:34:28 on dist-test-slave-pqr8"
server_key: "b9bca34410f3326da1d38e2981b28081"
server_key_iv: "7b4c5299d936b27d48f2fa5c32d89975"
server_key_version: "encryptionkey@0"
I20260927 21:34:28.034049 14477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:34:28.037062 14492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:28.038216 14477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:34:28.038373 14477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "b632fe78821b4594a2776f0672e7a5a1"
format_stamp: "Formatted at 2026-09-27 21:34:28 on dist-test-slave-pqr8"
server_key: "b9bca34410f3326da1d38e2981b28081"
server_key_iv: "7b4c5299d936b27d48f2fa5c32d89975"
server_key_version: "encryptionkey@0"
I20260927 21:34:28.038537 14477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:34:28.061749 14477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:34:28.065292 14477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:34:28.065603 14477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:34:28.066402 14477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:34:28.067533 14477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:34:28.067626 14477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:28.067754 14477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:34:28.067836 14477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:34:28.099064 14477 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:42841
I20260927 21:34:28.099099 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:42841 every 8 connection(s)
I20260927 21:34:28.100144 14477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:34:28.100682 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 14477
I20260927 21:34:28.100888 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:34:28.101423 3819 external_mini_cluster.cc:1428] Setting key 9396896e3ad918478bf9a403ab98aaab
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544868, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:34:28.118592 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.102589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:54621 (local address 127.3.186.254:41103)
0927 21:34:28.102773 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.102779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.104135 (+ 1356us) server_negotiation.cc:407] Connection header received
0927 21:34:28.105942 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.105946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.106003 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.106079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:28.107335 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.108045 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.109438 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.109919 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.110025 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.112984 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:28.112989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:28.112992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:28.113020 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:28.115722 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.116495 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.116503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.116506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.116664 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.117072 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.117079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.117082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.117320 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:28.117465 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.118072 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.118264 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:34:28.119340 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.103170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:41103 (local address 127.3.186.195:54621)
0927 21:34:28.103965 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:28.104010 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:28.105734 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:28.106257 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:28.106263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:28.106679 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:34:28.107207 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.107222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.108159 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.108163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.109322 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.109337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.110105 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.110110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.110279 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.110835 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:34:28.110854 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:34:28.112820 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:34:28.115852 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:28.115858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:28.115866 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:28.116385 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:28.116785 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:34:28.116791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:34:28.116794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:34:28.116958 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:34:28.117434 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:34:28.117442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:34:28.117754 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 21:34:28.117987 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 21:34:28.118392 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":307,"threads_started":1}
I20260927 21:34:28.120450 14605 heartbeater.cc:347] Connected to a master server at 127.3.186.254:41103
I20260927 21:34:28.120739 14605 heartbeater.cc:464] Registering TS with master...
I20260927 21:34:28.121354 14605 heartbeater.cc:511] Master 127.3.186.254:41103 requested a full tablet report, sending...
I20260927 21:34:28.122641 14149 ts_manager.cc:194] Registered new tserver with Master: b632fe78821b4594a2776f0672e7a5a1 (127.3.186.195:42841)
I20260927 21:34:28.123407 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:54621
I20260927 21:34:28.128494 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:34:28.143661 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.132750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.3.186.254:41103)
0927 21:34:28.132880 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.132884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.133380 (+ 496us) server_negotiation.cc:407] Connection header received
0927 21:34:28.133500 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.133504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.133602 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.133679 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:28.134864 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.135508 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.136634 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.136989 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.137074 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.139037 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:28.139041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:28.139044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:28.139065 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:28.141291 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.141888 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.141891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.141893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.142010 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.142399 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.142402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.142404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.142680 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:28.142840 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.143287 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.143412 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:34:28.147140 14149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:34:28.153297 14149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:34:28.174500 14338 heartbeater.cc:503] Master 127.3.186.254:41103 was elected leader, sending a full tablet report...
I20260927 21:34:28.178406 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.165061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.3.186.193:34881)
0927 21:34:28.165939 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.165949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.166542 (+ 593us) server_negotiation.cc:407] Connection header received
0927 21:34:28.166865 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.166877 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.167118 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.167271 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:34:28.168352 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.170375 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.173757 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.176397 (+ 2640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.176526 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.176785 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.177950 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.178098 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":570,"threads_started":1}
I20260927 21:34:28.178447 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.164489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34881 (local address 127.0.0.1:57370)
0927 21:34:28.166419 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:28.166440 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:28.166664 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:28.167592 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:28.167602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:28.167639 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:34:28.168112 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.168125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.170568 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.170575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.173558 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.173576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.177014 (+ 3438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.177033 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.177703 (+ 670us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.177763 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:34:28.177896 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 21:34:28.178057 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1776,"mutex_wait_us":395,"thread_start_us":1609,"threads_started":1}
I20260927 21:34:28.181209 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.163682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.3.186.194:36595)
0927 21:34:28.171136 (+ 7454us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.171144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.171169 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:34:28.171259 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.171266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.171478 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.171697 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:34:28.172845 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.174666 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.178705 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.180321 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.180471 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.180656 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.180818 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.180955 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7316,"thread_start_us":7213,"threads_started":1}
I20260927 21:34:28.181257 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.163616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:36595 (local address 127.0.0.1:59492)
0927 21:34:28.165311 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:28.165355 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:28.165585 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:28.171994 (+ 6409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:28.172002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:28.172035 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:34:28.172585 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.172599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.174827 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.174835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.177403 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.177422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.180499 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.180505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.180649 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.180679 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:34:28.180784 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:34:28.180886 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"mutex_wait_us":1,"thread_start_us":1316,"threads_started":1}
I20260927 21:34:28.182163 14273 tablet_service.cc:1467] Processing CreateTablet for tablet 0e78bee487234c4fa13c90b8ad96a69b (DEFAULT_TABLE table=test-table [id=c4871392a27f4701bfc3ad8444e2027a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:34:28.183458 14273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e78bee487234c4fa13c90b8ad96a69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:28.184850 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.167104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.3.186.195:42841)
0927 21:34:28.177603 (+ 10499us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.177608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.178176 (+ 568us) server_negotiation.cc:407] Connection header received
0927 21:34:28.178260 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.178267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.178886 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.179069 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:34:28.179599 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.180984 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.183303 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.184077 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.184166 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.184291 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.184559 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.184650 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10341,"thread_start_us":10220,"threads_started":1}
I20260927 21:34:28.184887 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.166611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:42841 (local address 127.0.0.1:49276)
0927 21:34:28.177950 (+ 11339us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:28.177965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:28.178075 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:28.179090 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:28.179095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:28.179118 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:34:28.179414 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.179424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.181150 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.181156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.183147 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.183161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.184252 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.184257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.184399 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.184425 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:34:28.184509 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:34:28.184623 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11200,"thread_start_us":11098,"threads_started":1}
I20260927 21:34:28.188097 14540 tablet_service.cc:1467] Processing CreateTablet for tablet 0e78bee487234c4fa13c90b8ad96a69b (DEFAULT_TABLE table=test-table [id=c4871392a27f4701bfc3ad8444e2027a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:34:28.188067 14407 tablet_service.cc:1467] Processing CreateTablet for tablet 0e78bee487234c4fa13c90b8ad96a69b (DEFAULT_TABLE table=test-table [id=c4871392a27f4701bfc3ad8444e2027a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:34:28.189302 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e78bee487234c4fa13c90b8ad96a69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:28.189301 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e78bee487234c4fa13c90b8ad96a69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:34:28.196832 14623 tablet_bootstrap.cc:492] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Bootstrap starting.
I20260927 21:34:28.199618 14623 tablet_bootstrap.cc:654] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:28.200611 14623 log.cc:824] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:28.203009 14626 tablet_bootstrap.cc:492] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Bootstrap starting.
I20260927 21:34:28.203361 14625 tablet_bootstrap.cc:492] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Bootstrap starting.
I20260927 21:34:28.204023 14623 tablet_bootstrap.cc:492] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: No bootstrap required, opened a new log
I20260927 21:34:28.204275 14623 ts_tablet_manager.cc:1397] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 21:34:28.205916 14625 tablet_bootstrap.cc:654] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:28.206324 14626 tablet_bootstrap.cc:654] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Neither blocks nor log segments found. Creating new log.
I20260927 21:34:28.206841 14625 log.cc:824] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:28.207362 14626 log.cc:824] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:34:28.209501 14625 tablet_bootstrap.cc:492] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: No bootstrap required, opened a new log
I20260927 21:34:28.209504 14626 tablet_bootstrap.cc:492] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: No bootstrap required, opened a new log
I20260927 21:34:28.209743 14625 ts_tablet_manager.cc:1397] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 21:34:28.209839 14626 ts_tablet_manager.cc:1397] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260927 21:34:28.214183 14623 raft_consensus.cc:348] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.214532 14623 raft_consensus.cc:374] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:28.214622 14623 raft_consensus.cc:729] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1c80cd3308479da4b21c601f291e41, State: Initialized, Role: FOLLOWER
I20260927 21:34:28.215171 14623 consensus_queue.cc:260] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.216646 14623 ts_tablet_manager.cc:1428] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260927 21:34:28.218135 14626 raft_consensus.cc:348] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.218431 14626 raft_consensus.cc:374] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:28.218551 14626 raft_consensus.cc:729] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b632fe78821b4594a2776f0672e7a5a1, State: Initialized, Role: FOLLOWER
I20260927 21:34:28.219087 14626 consensus_queue.cc:260] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.219570 14625 raft_consensus.cc:348] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.219919 14625 raft_consensus.cc:374] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:34:28.220001 14625 raft_consensus.cc:729] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fba8721f084feb9704586920504955, State: Initialized, Role: FOLLOWER
I20260927 21:34:28.220011 14605 heartbeater.cc:503] Master 127.3.186.254:41103 was elected leader, sending a full tablet report...
I20260927 21:34:28.220496 14625 consensus_queue.cc:260] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.220755 14626 ts_tablet_manager.cc:1428] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260927 21:34:28.222299 14625 ts_tablet_manager.cc:1428] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260927 21:34:28.222656 14472 heartbeater.cc:503] Master 127.3.186.254:41103 was elected leader, sending a full tablet report...
I20260927 21:34:28.224356 14629 raft_consensus.cc:482] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:34:28.224615 14629 raft_consensus.cc:504] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.226130 14629 leader_election.cc:288] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fba8721f084feb9704586920504955 (127.3.186.194:36595), b632fe78821b4594a2776f0672e7a5a1 (127.3.186.195:42841)
I20260927 21:34:28.233924 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.226794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:54295 (local address 127.3.186.194:36595)
0927 21:34:28.227006 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.227013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.227585 (+ 572us) server_negotiation.cc:407] Connection header received
0927 21:34:28.227703 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.227711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.227790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.227879 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:34:28.228503 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.229457 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.231855 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.233060 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.233204 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.233268 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.233595 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.233701 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:34:28.234022 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.226508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:36595 (local address 127.3.186.193:54295)
0927 21:34:28.227490 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:28.227503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:28.227600 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:28.227935 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:28.227939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:28.227964 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:34:28.228351 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.228361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.229606 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.229610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.231717 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.231731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.233231 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.233235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.233409 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.233441 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:34:28.233560 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:34:28.233686 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":1235,"threads_started":1}
I20260927 21:34:28.234051 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.226465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:38739 (local address 127.3.186.195:42841)
0927 21:34:28.226625 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.226631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.228161 (+ 1530us) server_negotiation.cc:407] Connection header received
0927 21:34:28.228307 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.228310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.228356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.228420 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:34:28.228869 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.229816 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.232042 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.233226 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.233353 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.233394 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.233705 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.233817 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:34:28.234366 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.226541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:42841 (local address 127.3.186.193:38739)
0927 21:34:28.228062 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0927 21:34:28.228079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:34:28.228198 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:34:28.228457 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:34:28.228461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:34:28.228486 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:34:28.228731 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.228737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.229923 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.229928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.231913 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:34:28.231922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.233402 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.233407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:34:28.233578 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.233604 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:34:28.233675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:34:28.233767 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"spinlock_wait_cycles":13696,"thread_start_us":659,"threads_started":1}
I20260927 21:34:28.234828 14427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e78bee487234c4fa13c90b8ad96a69b" candidate_uuid: "cd1c80cd3308479da4b21c601f291e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fba8721f084feb9704586920504955" is_pre_election: true
I20260927 21:34:28.235193 14427 raft_consensus.cc:2454] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1c80cd3308479da4b21c601f291e41 in term 0.
I20260927 21:34:28.235172 14560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e78bee487234c4fa13c90b8ad96a69b" candidate_uuid: "cd1c80cd3308479da4b21c601f291e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b632fe78821b4594a2776f0672e7a5a1" is_pre_election: true
I20260927 21:34:28.235679 14560 raft_consensus.cc:2454] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1c80cd3308479da4b21c601f291e41 in term 0.
I20260927 21:34:28.235824 14229 leader_election.cc:302] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7fba8721f084feb9704586920504955, cd1c80cd3308479da4b21c601f291e41; no voters:
I20260927 21:34:28.236181 14629 raft_consensus.cc:2781] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:34:28.236325 14629 raft_consensus.cc:482] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:34:28.236436 14629 raft_consensus.cc:3037] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:28.238394 14629 raft_consensus.cc:504] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.238950 14629 leader_election.cc:288] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [CANDIDATE]: Term 1 election: Requested vote from peers c7fba8721f084feb9704586920504955 (127.3.186.194:36595), b632fe78821b4594a2776f0672e7a5a1 (127.3.186.195:42841)
I20260927 21:34:28.239315 14427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e78bee487234c4fa13c90b8ad96a69b" candidate_uuid: "cd1c80cd3308479da4b21c601f291e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fba8721f084feb9704586920504955"
I20260927 21:34:28.239358 14560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e78bee487234c4fa13c90b8ad96a69b" candidate_uuid: "cd1c80cd3308479da4b21c601f291e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b632fe78821b4594a2776f0672e7a5a1"
I20260927 21:34:28.239493 14427 raft_consensus.cc:3037] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:28.239513 14560 raft_consensus.cc:3037] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:34:28.241488 14560 raft_consensus.cc:2454] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1c80cd3308479da4b21c601f291e41 in term 1.
I20260927 21:34:28.241741 14427 raft_consensus.cc:2454] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1c80cd3308479da4b21c601f291e41 in term 1.
I20260927 21:34:28.241904 14227 leader_election.cc:302] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b632fe78821b4594a2776f0672e7a5a1, cd1c80cd3308479da4b21c601f291e41; no voters:
I20260927 21:34:28.242268 14629 raft_consensus.cc:2781] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:34:28.242702 14629 raft_consensus.cc:686] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 1 LEADER]: Becoming Leader. State: Replica: cd1c80cd3308479da4b21c601f291e41, State: Running, Role: LEADER
I20260927 21:34:28.243183 14629 consensus_queue.cc:237] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } }
I20260927 21:34:28.246693 14149 catalog_manager.cc:5495] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd1c80cd3308479da4b21c601f291e41 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "cd1c80cd3308479da4b21c601f291e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd1c80cd3308479da4b21c601f291e41" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 } health_report { overall_health: UNKNOWN } } }
I20260927 21:34:28.272184 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.267473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.3.186.193:34881)
0927 21:34:28.267666 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.267674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.267765 (+ 91us) server_negotiation.cc:407] Connection header received
0927 21:34:28.267903 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.267907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.267977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.268086 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:34:28.268383 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.269084 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.270405 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.270781 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.270870 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.271257 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:34:28.271672 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:34:28.271777 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.271861 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.271915 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:34:28.279029 14427 raft_consensus.cc:1261] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 1 FOLLOWER]: Refusing update from remote peer cd1c80cd3308479da4b21c601f291e41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:34:28.279027 14560 raft_consensus.cc:1261] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 1 FOLLOWER]: Refusing update from remote peer cd1c80cd3308479da4b21c601f291e41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:34:28.279973 14629 consensus_queue.cc:1035] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fba8721f084feb9704586920504955" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:34:28.280304 14634 consensus_queue.cc:1035] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b632fe78821b4594a2776f0672e7a5a1" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:34:28.295073 14637 mvcc.cc:205] Tried to move back new op lower bound from 7334071780461957120 to 7334071780327067648. Current Snapshot: MvccSnapshot[applied={T|T < 7334071780461957120}]
I20260927 21:34:28.297417 14639 mvcc.cc:205] Tried to move back new op lower bound from 7334071780461957120 to 7334071780327067648. Current Snapshot: MvccSnapshot[applied={T|T < 7334071780461957120}]
I20260927 21:34:28.308190 14148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:34:28.308450 14148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:34:28.311504 14148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 261554f320f94701893553d0aa5a7b2d: Sending DeleteTablet for 3 replicas of tablet 0e78bee487234c4fa13c90b8ad96a69b
I20260927 21:34:28.312506 14407 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e78bee487234c4fa13c90b8ad96a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:34:28 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59492
I20260927 21:34:28.312568 14273 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e78bee487234c4fa13c90b8ad96a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:34:28 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:57370
I20260927 21:34:28.313112 14540 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e78bee487234c4fa13c90b8ad96a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:34:28 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:49276
I20260927 21:34:28.315310 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 14210
I20260927 21:34:28.346220 14647 tablet_replica.cc:330] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: stopping tablet replica
I20260927 21:34:28.346608 14646 tablet_replica.cc:330] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: stopping tablet replica
I20260927 21:34:28.346597 14647 raft_consensus.cc:2229] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:34:28.347008 14647 raft_consensus.cc:2258] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:34:28.347045 14646 raft_consensus.cc:2229] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:34:28.346040 14648 tablet_replica.cc:330] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: stopping tablet replica
I20260927 21:34:28.347759 14646 raft_consensus.cc:2258] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:34:28.349301 14647 ts_tablet_manager.cc:1905] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:34:28.349623 14646 ts_tablet_manager.cc:1905] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:34:28.353925 14648 raft_consensus.cc:2229] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:34:28.356043 14647 ts_tablet_manager.cc:1918] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:34:28.356181 14647 log.cc:1196] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/0e78bee487234c4fa13c90b8ad96a69b
I20260927 21:34:28.356671 14647 ts_tablet_manager.cc:1939] T 0e78bee487234c4fa13c90b8ad96a69b P c7fba8721f084feb9704586920504955: Deleting consensus metadata
I20260927 21:34:28.357105 14648 raft_consensus.cc:2258] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:34:28.358064 14646 ts_tablet_manager.cc:1918] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:34:28.358282 14646 log.cc:1196] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/0e78bee487234c4fa13c90b8ad96a69b
I20260927 21:34:28.358328 14136 catalog_manager.cc:4841] TS c7fba8721f084feb9704586920504955 (127.3.186.194:36595): tablet 0e78bee487234c4fa13c90b8ad96a69b (table test-table [id=c4871392a27f4701bfc3ad8444e2027a]) successfully deleted
I20260927 21:34:28.358898 14646 ts_tablet_manager.cc:1939] T 0e78bee487234c4fa13c90b8ad96a69b P cd1c80cd3308479da4b21c601f291e41: Deleting consensus metadata
I20260927 21:34:28.360035 14648 ts_tablet_manager.cc:1905] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:34:28.360495 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.316795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.3.186.193:34881)
0927 21:34:28.317002 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.317010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.338193 (+ 21183us) server_negotiation.cc:407] Connection header received
0927 21:34:28.338635 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.338644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.338782 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.338875 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:28.341001 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.342546 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.346107 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.346809 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.346927 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.351866 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:28.351880 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:28.351893 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:28.351939 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:28.355970 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.356846 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.356854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.356861 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.357047 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.357740 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.357746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.357750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.358189 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:28.358493 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.359789 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.360022 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:34:28.360500 14133 catalog_manager.cc:4841] TS cd1c80cd3308479da4b21c601f291e41 (127.3.186.193:34881): tablet 0e78bee487234c4fa13c90b8ad96a69b (table test-table [id=c4871392a27f4701bfc3ad8444e2027a]) successfully deleted
I20260927 21:34:28.361075 14333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:28.366007 14648 ts_tablet_manager.cc:1918] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:34:28.366158 14648 log.cc:1196] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/0e78bee487234c4fa13c90b8ad96a69b
I20260927 21:34:28.366598 14648 ts_tablet_manager.cc:1939] T 0e78bee487234c4fa13c90b8ad96a69b P b632fe78821b4594a2776f0672e7a5a1: Deleting consensus metadata
I20260927 21:34:28.367760 14134 catalog_manager.cc:4841] TS b632fe78821b4594a2776f0672e7a5a1 (127.3.186.195:42841): tablet 0e78bee487234c4fa13c90b8ad96a69b (table test-table [id=c4871392a27f4701bfc3ad8444e2027a]) successfully deleted
I20260927 21:34:28.456554 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 14210
I20260927 21:34:28.474501 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 14344
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:34:28.488047 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.475234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.3.186.194:36595)
0927 21:34:28.475397 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.475401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.475419 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:34:28.475480 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.475484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.475541 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.475618 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:28.476695 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.477351 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.478541 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.478912 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.478999 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.482816 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:28.482828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:28.482843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:28.482876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:28.485479 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.486126 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.486143 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.486145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.486259 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.486597 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.486602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.486606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.486947 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:28.487229 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.487518 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.487716 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:34:28.488682 14467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:28.572062 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 14344
W20260927 21:34:28.586001 14180 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.3.186.254:41103
I20260927 21:34:28.594170 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 14477
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:34:28.608959 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:34:28.594874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.3.186.195:42841)
0927 21:34:28.595043 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:34:28.595048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:34:28.595068 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:34:28.595191 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:34:28.595196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:34:28.595278 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:34:28.595387 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:34:28.596504 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.597548 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.598667 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:34:28.599265 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:34:28.599373 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:34:28.603638 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:34:28.603654 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:34:28.603673 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:34:28.603707 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:34:28.606798 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.607384 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.607390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.607392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.607519 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:34:28.607824 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:34:28.607827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:34:28.607828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:34:28.608164 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:34:28.608362 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:34:28.608588 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 21:34:28.608726 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:34:28.609618 14600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:28.694842 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 14477
I20260927 21:34:28.710741 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 14117
I20260927 21:34:28.711850 14179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:34:28.809990 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 14117
2026-09-27T21:34:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3909 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14656](info): set up 2 sockets
Sep 27 21:34:28 dist-test-slave-pqr8 krb5kdc[14656](info): commencing operation
krb5kdc: starting...
W20260927 21:34:29.882208 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:34:30 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:34:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/rangeradmin_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/rangeradmin_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/rangerlookup_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/rangerlookup_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
I20260927 21:34:30.150527 3819 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWw8W86/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/postgres -l logfile start
2026-09-27 21:34:32.446 UTC [14686] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:34:32.447 UTC [14686] LOG: listening on IPv4 address "127.3.186.212", port 42087
2026-09-27 21:34:32.451 UTC [14686] LOG: listening on Unix socket "/tmp/.s.PGSQL.42087"
2026-09-27 21:34:32.472 UTC [14689] LOG: database system was shut down at 2026-09-27 21:34:31 UTC
2026-09-27 21:34:32.475 UTC [14686] LOG: database system is ready to accept connections
I20260927 21:34:33.405972 3819 mini_postgres.cc:96] Postgres bound to 42087
2026-09-27 21:34:33.422 UTC [14697] FATAL: database "slave" does not exist
127.3.186.212:42087 - accepting connections
I20260927 21:34:33.422885 3819 mini_ranger.cc:161] Starting Ranger...
I20260927 21:34:33.445427 3819 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:34:33.886013 3819 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:34:33.886143 3819 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-admin
2026-09-27 21:34:34,173 [I] DB FLAVOR :POSTGRES
2026-09-27 21:34:34,174 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:34:34,174 [I] Checking connection..
2026-09-27 21:34:34,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:34:34,428 [I] Checking connection passed.
2026-09-27 21:34:34,428 [I] --------- Verifying version history table ---------
2026-09-27 21:34:34,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:34:34,672 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:34:34,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:34:34,916 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:34:34,916 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:34:34,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:34:35.145 UTC [14761] WARNING: there is no transaction in progress
2026-09-27 21:34:35,151 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:34:35,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:34:35,386 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:34:35,386 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:34:35,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:34:35,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:34:35,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pqr8.c.gcp-upstream.internal','N') ;"
2026-09-27 21:34:36,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:34:36,281 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:34:36,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:34:36,518 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:34:36,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:34:36,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:34:36,945 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:34:36,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:34:37.813 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.828 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.843 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.858 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.875 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.891 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.941 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.947 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.960 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.970 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.978 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.991 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:37.999 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:38.005 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:38.011 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:38.017 UTC [14869] WARNING: there is no transaction in progress
2026-09-27 21:34:38,590 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:34:38,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:34:38,826 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:34:38,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:34:39,039 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:34:39,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:34:39,259 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:34:39,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:34:39,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:34:39,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pqr8.c.gcp-upstream.internal';"
2026-09-27 21:34:39,968 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:34:39,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:34:40,188 [I] DB_PATCHES have already been applied
I20260927 21:34:40.194804 3819 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWw8W86/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:34:40.194887 3819 mini_ranger.cc:193] Using host: 127.3.186.212
I20260927 21:34:40.208626 3819 mini_ranger.cc:227] Ranger admin URL: http://127.3.186.212:53401
Sep 27, 2026 9:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:34:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:34:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:34:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:34:41.320807 3819 mini_ranger.cc:160] Time spent starting Ranger: real 7.898s user 0.002s sys 0.028s
Sep 27, 2026 9:35:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:35:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 9:35:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:35:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:35:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:35:01.893952 3819 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/rangerkms_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/rangerkms_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.186.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.186.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:35:02 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544902, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:35:02.022384 3819 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:35:02.038138 3819 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:35:02.843883 3819 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:35:02.844040 3819 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms
I20260927 21:35:02.844173 3819 mini_ranger_kms.cc:227] Using postgres at 127.3.186.212:42087
2026-09-27 21:35:03,237 [I] DB FLAVOR :POSTGRES
2026-09-27 21:35:03,237 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:35:03,237 [I] Checking connection
2026-09-27 21:35:03,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:35:03,463 [I] connection success
2026-09-27 21:35:03,464 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:35:03,464 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:35:03,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:35:03,700 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:35:03,700 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:35:03,700 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:35:03,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:42087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:35:03,941 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:35:04.332966 3819 mini_ranger_kms.cc:310] Created kms service
I20260927 21:35:04.513688 3819 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:35:04.554378 3819 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:35:04 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544902, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:04.948208 3819 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:35:04.948527 3819 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:35:04.948586 3819 mini_ranger_kms.cc:240] Using host: 127.3.186.212
I20260927 21:35:04.967495 3819 mini_ranger_kms.cc:276] Ranger KMS PID: 15049
I20260927 21:35:04.967653 3819 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.186.212:34067
Sep 27, 2026 9:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:35:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:35:07.261554 3819 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.239s user 0.137s sys 0.272s
I20260927 21:35:07.261782 3819 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:35:07.262056 3819 mini_ranger_kms.cc:401] 127.3.186.212:34067/kms/v1/keys
Sep 27 21:35:07 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544907, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.186.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:35:07 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544907, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.186.212@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
Sep 27 21:35:08 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544907, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.186.212@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
Sep 27 21:35:10 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:35:10.192747 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44863
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44863
--ranger_config_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:34067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:35:10.504906 15100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:10.505153 15100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:10.505280 15100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:10.511869 15100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:35:10.511952 15100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:10.512036 15100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:10.512116 15100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:35:10.512188 15100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:35:10.520833 15100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.186.212:34067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44863
--ranger_config_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:10.521931 15100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:10.523491 15100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:10.534240 15108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:10.534279 15106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:10.534291 15105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:10.534873 15100 server_base.cc:1034] running on GCE node
I20260927 21:35:10.535586 15100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:10.536814 15100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:10.538017 15100 hybrid_clock.cc:648] HybridClock initialized: now 1790544910537978 us; error 48 us; skew 500 ppm
I20260927 21:35:10.538322 15100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:10 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:10.548195 15100 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:35:10.549494 15100 webserver.cc:460] Webserver started at http://127.3.186.254:43657/ using document root <none> and password file <none>
I20260927 21:35:10.550062 15100 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:10.550148 15100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:10.550366 15100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:10 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:10.597469 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "a5104369e5234f628e19655c95c715e5"
format_stamp: "Formatted at 2026-09-27 21:35:10 on dist-test-slave-pqr8"
server_key: "52d2d7c7e29e3d5c01f70f0d70c5e424"
server_key_iv: "e4f49dd8b7463688758ec3573337291a"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:10.598412 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "a5104369e5234f628e19655c95c715e5"
format_stamp: "Formatted at 2026-09-27 21:35:10 on dist-test-slave-pqr8"
server_key: "52d2d7c7e29e3d5c01f70f0d70c5e424"
server_key_iv: "e4f49dd8b7463688758ec3573337291a"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:10.603384 15100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:35:10.621981 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:10.623332 15100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:35:10.623492 15100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "a5104369e5234f628e19655c95c715e5"
format_stamp: "Formatted at 2026-09-27 21:35:10 on dist-test-slave-pqr8"
server_key: "52d2d7c7e29e3d5c01f70f0d70c5e424"
server_key_iv: "e4f49dd8b7463688758ec3573337291a"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:10.623687 15100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:10.645591 15100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:10.648790 15100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:10.649062 15100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:10.666676 15100 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44863
I20260927 21:35:10.666706 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44863 every 8 connection(s)
I20260927 21:35:10.667835 15100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:35:10.671222 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:35:10.672857 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 15100
I20260927 21:35:10.673039 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:35:10.673858 3819 external_mini_cluster.cc:1428] Setting key 78f8fdedc8b417762bdd25275aefce0e
I20260927 21:35:10.684844 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5: Bootstrap starting.
Sep 27 21:35:10 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:10.688092 15167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5: Neither blocks nor log segments found. Creating new log.
I20260927 21:35:10.689110 15167 log.cc:824] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:35:10.692104 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5: No bootstrap required, opened a new log
I20260927 21:35:10.695485 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:10.675710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.3.186.254:44863)
0927 21:35:10.676458 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:10.676468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:10.676507 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:35:10.678409 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:10.678431 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:10.678765 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:10.679032 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:10.680485 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:10.681742 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:10.683443 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:10.684069 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:10.684193 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:10.688300 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:10.688307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:10.688343 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:10.688374 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:10.691536 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:10.692164 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:10.692176 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:10.692182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:10.692361 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:10.692702 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:10.692708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:10.692711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:10.693131 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:10.693414 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:10.693873 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 21:35:10.694252 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":371,"threads_started":1}
I20260927 21:35:10.699674 15167 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5104369e5234f628e19655c95c715e5" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44863 } }
I20260927 21:35:10.699970 15167 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:35:10.700035 15167 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5104369e5234f628e19655c95c715e5, State: Initialized, Role: FOLLOWER
I20260927 21:35:10.700497 15167 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5104369e5234f628e19655c95c715e5" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44863 } }
I20260927 21:35:10.700685 15167 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:35:10.700770 15167 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:35:10.700889 15167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:35:10.702692 15167 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5104369e5234f628e19655c95c715e5" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44863 } }
I20260927 21:35:10.703133 15167 leader_election.cc:302] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5104369e5234f628e19655c95c715e5; no voters:
I20260927 21:35:10.703626 15167 leader_election.cc:288] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:35:10.703751 15172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:35:10.704840 15172 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [term 1 LEADER]: Becoming Leader. State: Replica: a5104369e5234f628e19655c95c715e5, State: Running, Role: LEADER
I20260927 21:35:10.705389 15172 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5104369e5234f628e19655c95c715e5" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44863 } }
I20260927 21:35:10.706063 15167 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:35:10.708386 15174 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5104369e5234f628e19655c95c715e5. Latest consensus state: current_term: 1 leader_uuid: "a5104369e5234f628e19655c95c715e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5104369e5234f628e19655c95c715e5" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44863 } } }
I20260927 21:35:10.708385 15173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5104369e5234f628e19655c95c715e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5104369e5234f628e19655c95c715e5" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44863 } } }
I20260927 21:35:10.708863 15174 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:35:10.708889 15173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:35:10.713289 15180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:35:10.714398 15167 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:35:10.725698 15180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:35:10.744884 15180 catalog_manager.cc:1347] Generated new cluster ID: e4902575b3674cc8871c84ed6d55b011
I20260927 21:35:10.744988 15180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:35:10.766199 15180 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:35:10.768016 15180 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:35:10.779074 15180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5: Generated new TSK 0
I20260927 21:35:10.779995 15180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:35:11.461684 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:34067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44863
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:35:11 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544911, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:35:11.881726 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:11.882048 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:11.882195 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:11.892135 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:11.892226 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:11.892380 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:35:11.904918 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.186.212:34067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:11.906252 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:11.907575 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:11.928572 15222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:11.929831 15223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:11.935984 15212 server_base.cc:1034] running on GCE node
W20260927 21:35:11.936580 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:11.937186 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:11.938208 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:11.939424 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790544911939369 us; error 57 us; skew 500 ppm
I20260927 21:35:11.939709 15212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:11 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544911, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:11.943008 15212 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:35:11.944595 15212 webserver.cc:460] Webserver started at http://127.3.186.193:41803/ using document root <none> and password file <none>
I20260927 21:35:11.945240 15212 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:11.945359 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:11.945717 15212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:11 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544911, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:11.964906 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "a4ec6910431b40dc81dac9fd70d106c5"
format_stamp: "Formatted at 2026-09-27 21:35:11 on dist-test-slave-pqr8"
server_key: "be9b1e6815d9bcac107cfbded2f180b2"
server_key_iv: "11844b2bf3582859967c60b1bd376e37"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:11.965951 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "a4ec6910431b40dc81dac9fd70d106c5"
format_stamp: "Formatted at 2026-09-27 21:35:11 on dist-test-slave-pqr8"
server_key: "be9b1e6815d9bcac107cfbded2f180b2"
server_key_iv: "11844b2bf3582859967c60b1bd376e37"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:11.991142 15212 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.011s sys 0.013s
I20260927 21:35:12.026014 15232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:12.027751 15212 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.008s sys 0.008s
I20260927 21:35:12.027976 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "a4ec6910431b40dc81dac9fd70d106c5"
format_stamp: "Formatted at 2026-09-27 21:35:11 on dist-test-slave-pqr8"
server_key: "be9b1e6815d9bcac107cfbded2f180b2"
server_key_iv: "11844b2bf3582859967c60b1bd376e37"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:12.028363 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:12.070582 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:12.075359 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:12.075709 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:12.082824 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:35:12.091040 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:35:12.091178 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 21:35:12.091367 15212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:35:12.091485 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:35:12 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544911, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:12.452687 15212 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:46113
I20260927 21:35:12.452725 15344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:46113 every 8 connection(s)
I20260927 21:35:12.454391 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:35:12.464604 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 15212
I20260927 21:35:12.464785 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:35:12.465504 3819 external_mini_cluster.cc:1428] Setting key 94b134423ff396863a56d1f4f8dbaa98
Sep 27 21:35:12 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544911, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:12.521687 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:12.459003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:44815 (local address 127.3.186.254:44863)
0927 21:35:12.490640 (+ 31637us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:12.490646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:12.490669 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:35:12.501980 (+ 11311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:12.501984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:12.502073 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:12.502178 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:12.505091 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:12.506342 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:12.508929 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:12.509659 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:12.509768 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:12.515047 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:12.515066 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:12.515070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:12.515121 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:12.518199 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:12.519253 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:12.519257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:12.519261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:12.519388 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:12.519882 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:12.519887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:12.519890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:12.520183 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:12.520284 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:12.521137 (+ 853us) server_negotiation.cc:299] Negotiation successful
0927 21:35:12.521337 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31516,"thread_start_us":31660,"threads_started":1}
I20260927 21:35:12.523757 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:12.460018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44863 (local address 127.3.186.193:44815)
0927 21:35:12.487387 (+ 27369us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:12.487436 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:12.501528 (+ 14092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:12.503374 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:12.503388 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:12.504050 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:35:12.504904 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:12.504932 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:12.506535 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:12.506542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:12.508766 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:12.508783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:12.510067 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:12.510074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:12.510355 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:12.511135 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:35:12.511169 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:35:12.514767 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:35:12.518347 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:12.518355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:12.518371 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:12.518832 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:12.519533 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:12.519537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:12.519541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:12.519755 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:12.520380 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:35:12.520387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:35:12.520737 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 21:35:12.522146 (+ 1409us) client_negotiation.cc:240] Negotiation successful
0927 21:35:12.522575 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27056,"thread_start_us":29409,"threads_started":1}
I20260927 21:35:12.525674 15345 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44863
Sep 27 21:35:12 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544911, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:12.525969 15345 heartbeater.cc:464] Registering TS with master...
I20260927 21:35:12.526670 15345 heartbeater.cc:511] Master 127.3.186.254:44863 requested a full tablet report, sending...
I20260927 21:35:12.529235 15132 ts_manager.cc:194] Registered new tserver with Master: a4ec6910431b40dc81dac9fd70d106c5 (127.3.186.193:46113)
I20260927 21:35:12.531201 15132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:44815
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:35:12.580036 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:34067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44863
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:35:13.035137 15353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:13.035494 15353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:13.035717 15353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:13.046139 15353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:13.046274 15353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:13.046489 15353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:35:13.060988 15353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.186.212:34067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:13.062433 15353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:13.064015 15353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:13.079241 15365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:13.080948 15366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:13.082304 15368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:13.082988 15353 server_base.cc:1034] running on GCE node
I20260927 21:35:13.083590 15353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:13.084511 15353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:13.085704 15353 hybrid_clock.cc:648] HybridClock initialized: now 1790544913085653 us; error 47 us; skew 500 ppm
I20260927 21:35:13.085984 15353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:13 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:13.089476 15353 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:35:13.091557 15353 webserver.cc:460] Webserver started at http://127.3.186.194:45885/ using document root <none> and password file <none>
I20260927 21:35:13.092192 15353 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:13.092317 15353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:13.092615 15353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:13 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:13.108995 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450"
format_stamp: "Formatted at 2026-09-27 21:35:13 on dist-test-slave-pqr8"
server_key: "92010290afa8b28c47beecbf5d77f0b9"
server_key_iv: "a1dfe1d8606e9b52162f84d3efd52c9a"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:13.110050 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450"
format_stamp: "Formatted at 2026-09-27 21:35:13 on dist-test-slave-pqr8"
server_key: "92010290afa8b28c47beecbf5d77f0b9"
server_key_iv: "a1dfe1d8606e9b52162f84d3efd52c9a"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:13.116865 15353 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 21:35:13.138520 15375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:13.140173 15353 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.006s
I20260927 21:35:13.140343 15353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450"
format_stamp: "Formatted at 2026-09-27 21:35:13 on dist-test-slave-pqr8"
server_key: "92010290afa8b28c47beecbf5d77f0b9"
server_key_iv: "a1dfe1d8606e9b52162f84d3efd52c9a"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:13.140558 15353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:13.157284 15353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:13.163031 15353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:13.163338 15353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:13.164408 15353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:35:13.166214 15353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:35:13.166309 15353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:13.166486 15353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:35:13.166558 15353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:13.222434 15353 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:39745
I20260927 21:35:13.222456 15487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:39745 every 8 connection(s)
I20260927 21:35:13.223712 15353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:35:13.230017 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 15353
I20260927 21:35:13.230170 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:35:13.230654 3819 external_mini_cluster.cc:1428] Setting key b82b28ba858298a66d94c695775dda93
Sep 27 21:35:13 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:13.244380 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.226718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:57399 (local address 127.3.186.254:44863)
0927 21:35:13.227224 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.227230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.228185 (+ 955us) server_negotiation.cc:407] Connection header received
0927 21:35:13.229979 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.229985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.230067 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.230179 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:13.231532 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.232557 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.234174 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.234567 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.234695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.237906 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:13.237912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:13.237916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:13.237953 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:13.240448 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:13.241346 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:13.241351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:13.241355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:13.241500 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:13.242029 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:13.242034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:13.242038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:13.242395 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:13.242544 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.243329 (+ 785us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.243561 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":233,"threads_started":1}
I20260927 21:35:13.245054 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.227257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44863 (local address 127.3.186.194:57399)
0927 21:35:13.228025 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.228068 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.229723 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.230407 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.230414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.230775 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:35:13.231379 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.231403 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.232685 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.232690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.234010 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.234025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.234771 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.234777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.235005 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.235676 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:35:13.235696 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:35:13.237721 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:35:13.240575 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:13.240584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:13.240597 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:13.241203 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:13.241635 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:13.241642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:13.241646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:13.241908 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:13.242528 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:35:13.242539 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:35:13.242933 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.243201 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.243711 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
I20260927 21:35:13.246403 15488 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44863
I20260927 21:35:13.246676 15488 heartbeater.cc:464] Registering TS with master...
I20260927 21:35:13.247502 15488 heartbeater.cc:511] Master 127.3.186.254:44863 requested a full tablet report, sending...
I20260927 21:35:13.249076 15132 ts_manager.cc:194] Registered new tserver with Master: 2b7143df8a0942b6bdd4cd7fbc2e6450 (127.3.186.194:39745)
I20260927 21:35:13.250342 15132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:57399
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:35:13.280509 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:34067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44863
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:35:13.534719 15345 heartbeater.cc:503] Master 127.3.186.254:44863 was elected leader, sending a full tablet report...
W20260927 21:35:13.583150 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:13.583382 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:13.583521 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:13.590467 15494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:13.590590 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:13.590775 15494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:35:13.600282 15494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.186.212:34067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:13.601328 15494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:13.602465 15494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:13.614646 15499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:13.614660 15500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:13.614908 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:13.615973 15494 server_base.cc:1034] running on GCE node
I20260927 21:35:13.616535 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:13.617358 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:13.618539 15494 hybrid_clock.cc:648] HybridClock initialized: now 1790544913618480 us; error 49 us; skew 500 ppm
I20260927 21:35:13.618846 15494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:13 dist-test-slave-pqr8 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:13.621667 15494 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:35:13.622869 15494 webserver.cc:460] Webserver started at http://127.3.186.195:33767/ using document root <none> and password file <none>
I20260927 21:35:13.623405 15494 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:13.623492 15494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:13.623741 15494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:13 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:13.638113 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "c30b2282b91b4e91b2cb10186deae4ea"
format_stamp: "Formatted at 2026-09-27 21:35:13 on dist-test-slave-pqr8"
server_key: "06c57bd89c469db65845049a8375914b"
server_key_iv: "e7fd851776a068e74ab4c57a27e77006"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:13.638847 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "c30b2282b91b4e91b2cb10186deae4ea"
format_stamp: "Formatted at 2026-09-27 21:35:13 on dist-test-slave-pqr8"
server_key: "06c57bd89c469db65845049a8375914b"
server_key_iv: "e7fd851776a068e74ab4c57a27e77006"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:13.643976 15494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:35:13.658205 15509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:13.659471 15494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:35:13.659634 15494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "c30b2282b91b4e91b2cb10186deae4ea"
format_stamp: "Formatted at 2026-09-27 21:35:13 on dist-test-slave-pqr8"
server_key: "06c57bd89c469db65845049a8375914b"
server_key_iv: "e7fd851776a068e74ab4c57a27e77006"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:13.659811 15494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:13.681816 15494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:13.685218 15494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:13.685478 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:13.686398 15494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:35:13.687582 15494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:35:13.687686 15494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:13.687834 15494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:35:13.687896 15494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:13.725510 15494 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:38903
I20260927 21:35:13.725616 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:38903 every 8 connection(s)
I20260927 21:35:13.726686 15494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:35:13.727301 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 15494
I20260927 21:35:13.727458 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:35:13.727982 3819 external_mini_cluster.cc:1428] Setting key 2cef51f2b66cb79c726f2eb0a95fbb61
Sep 27 21:35:13 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544913, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:13.749868 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.729991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:36757 (local address 127.3.186.254:44863)
0927 21:35:13.730157 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.730165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.732571 (+ 2406us) server_negotiation.cc:407] Connection header received
0927 21:35:13.735380 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.735384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.735519 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.735626 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:13.737298 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.738013 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.740115 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.740492 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.740577 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.744362 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:13.744366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:13.744369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:13.744396 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:13.747002 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:13.747760 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:13.747764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:13.747766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:13.747877 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:13.748346 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:13.748349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:13.748351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:13.748572 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:13.748673 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.749421 (+ 748us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.749652 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:35:13.751015 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.730390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44863 (local address 127.3.186.195:36757)
0927 21:35:13.732430 (+ 2040us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.732476 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.735054 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.735874 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.735885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.736402 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:35:13.737133 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.737168 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.738156 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.738161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.739990 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.740004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.740689 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.740696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.740962 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.741610 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:35:13.741635 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:35:13.744172 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:35:13.747128 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:13.747136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:13.747148 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:13.747653 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:13.747983 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:13.747988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:13.747991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:13.748210 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:13.748689 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:35:13.748696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:35:13.749049 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.749335 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.749820 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":1076,"threads_started":1}
I20260927 21:35:13.752365 15622 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44863
I20260927 21:35:13.752666 15622 heartbeater.cc:464] Registering TS with master...
I20260927 21:35:13.753432 15622 heartbeater.cc:511] Master 127.3.186.254:44863 requested a full tablet report, sending...
I20260927 21:35:13.754972 15132 ts_manager.cc:194] Registered new tserver with Master: c30b2282b91b4e91b2cb10186deae4ea (127.3.186.195:38903)
I20260927 21:35:13.756021 15132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:36757
I20260927 21:35:13.767159 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:35:13.785195 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.771501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.3.186.254:44863)
0927 21:35:13.771646 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.771651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.772091 (+ 440us) server_negotiation.cc:407] Connection header received
0927 21:35:13.772217 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.772221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.772297 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.772383 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:13.773695 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.774709 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.775965 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.776526 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.776639 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.778787 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:13.778794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:13.778798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:13.778831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:13.781921 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:13.782842 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:13.782847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:13.782851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:13.782997 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:13.783419 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:13.783424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:13.783427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:13.783754 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:13.784029 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.784511 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.784771 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:35:13.790901 15132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:35:13.801106 15132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:35:13.844996 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.813203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:46113 (local address 127.0.0.1:49464)
0927 21:35:13.818554 (+ 5351us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.818601 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.818778 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.837646 (+ 18868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.837653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.837705 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:35:13.838133 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.838143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.839886 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.839893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.842414 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.842432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.843968 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.843974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.844212 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.844291 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.844406 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.844611 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5053,"mutex_wait_us":31,"thread_start_us":4904,"threads_started":1}
I20260927 21:35:13.845050 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.813706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.3.186.193:46113)
0927 21:35:13.837066 (+ 23360us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.837076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.837104 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:35:13.837194 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.837201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.837413 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.837649 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:35:13.838322 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.839765 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.842555 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.843753 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.843886 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.844099 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.844561 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.844712 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23178,"thread_start_us":23028,"threads_started":1}
I20260927 21:35:13.846076 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.815661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.3.186.194:39745)
0927 21:35:13.835238 (+ 19577us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.835250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.835878 (+ 628us) server_negotiation.cc:407] Connection header received
0927 21:35:13.835978 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.835985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.836241 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.836421 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:35:13.837194 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.838825 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.842097 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.843505 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.843606 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.843794 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.845614 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.845776 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18173,"thread_start_us":18249,"threads_started":1}
I20260927 21:35:13.846477 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.813203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:38903 (local address 127.0.0.1:46386)
0927 21:35:13.829960 (+ 16757us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.829980 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.830122 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.838768 (+ 8646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.838774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.838799 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:35:13.839309 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.839321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.841948 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.841953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.843905 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.843923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.845766 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.845773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.845931 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.845956 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.846033 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.846133 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16614,"mutex_wait_us":28,"thread_start_us":16430,"threads_started":1}
I20260927 21:35:13.847133 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.815034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:39745 (local address 127.0.0.1:38322)
0927 21:35:13.835622 (+ 20588us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.835638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.835749 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.836601 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.836610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.836642 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:35:13.837013 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.837025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.839043 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.839050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.841929 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.841942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.844006 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.844015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.844287 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.844320 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.844412 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.846453 (+ 2041us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20495,"mutex_wait_us":65,"thread_start_us":20384,"threads_started":1}
I20260927 21:35:13.847553 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.813210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.3.186.195:38903)
0927 21:35:13.838262 (+ 25052us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.838271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.838298 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:35:13.838375 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.838381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.838585 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.839515 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:35:13.839772 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.841781 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.844098 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.845540 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.846797 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.847037 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.847098 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.847248 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24905,"thread_start_us":24767,"threads_started":1}
I20260927 21:35:13.850402 15280 tablet_service.cc:1467] Processing CreateTablet for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 (DEFAULT_TABLE table=test-table [id=791dedbcd9684dae9c1291309d3c5108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:35:13.850775 15423 tablet_service.cc:1467] Processing CreateTablet for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 (DEFAULT_TABLE table=test-table [id=791dedbcd9684dae9c1291309d3c5108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:35:13.850775 15557 tablet_service.cc:1467] Processing CreateTablet for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 (DEFAULT_TABLE table=test-table [id=791dedbcd9684dae9c1291309d3c5108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:35:13.851634 15280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:35:13.852022 15557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:35:13.852021 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:35:13.870561 15642 tablet_bootstrap.cc:492] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Bootstrap starting.
I20260927 21:35:13.875103 15642 tablet_bootstrap.cc:654] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Neither blocks nor log segments found. Creating new log.
I20260927 21:35:13.875967 15640 tablet_bootstrap.cc:492] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Bootstrap starting.
I20260927 21:35:13.880261 15640 tablet_bootstrap.cc:654] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Neither blocks nor log segments found. Creating new log.
I20260927 21:35:13.881187 15642 log.cc:824] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Log is configured to *not* fsync() on all Append() calls
I20260927 21:35:13.881917 15641 tablet_bootstrap.cc:492] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Bootstrap starting.
I20260927 21:35:13.882062 15640 log.cc:824] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:35:13.885285 15640 tablet_bootstrap.cc:492] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: No bootstrap required, opened a new log
I20260927 21:35:13.885649 15640 ts_tablet_manager.cc:1397] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 21:35:13.885792 15641 tablet_bootstrap.cc:654] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Neither blocks nor log segments found. Creating new log.
I20260927 21:35:13.886922 15641 log.cc:824] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Log is configured to *not* fsync() on all Append() calls
I20260927 21:35:13.892555 15642 tablet_bootstrap.cc:492] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: No bootstrap required, opened a new log
I20260927 21:35:13.892827 15642 ts_tablet_manager.cc:1397] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.008s
I20260927 21:35:13.895805 15641 tablet_bootstrap.cc:492] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: No bootstrap required, opened a new log
I20260927 21:35:13.896072 15641 ts_tablet_manager.cc:1397] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 21:35:13.897229 15640 raft_consensus.cc:348] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.897665 15640 raft_consensus.cc:374] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:35:13.897780 15640 raft_consensus.cc:729] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ec6910431b40dc81dac9fd70d106c5, State: Initialized, Role: FOLLOWER
I20260927 21:35:13.898448 15640 consensus_queue.cc:260] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.904121 15642 raft_consensus.cc:348] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.904699 15642 raft_consensus.cc:374] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:35:13.904847 15642 raft_consensus.cc:729] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b2282b91b4e91b2cb10186deae4ea, State: Initialized, Role: FOLLOWER
I20260927 21:35:13.905579 15642 consensus_queue.cc:260] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.906410 15640 ts_tablet_manager.cc:1428] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260927 21:35:13.907191 15641 raft_consensus.cc:348] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.907621 15641 raft_consensus.cc:374] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:35:13.907742 15641 raft_consensus.cc:729] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7143df8a0942b6bdd4cd7fbc2e6450, State: Initialized, Role: FOLLOWER
I20260927 21:35:13.908804 15622 heartbeater.cc:503] Master 127.3.186.254:44863 was elected leader, sending a full tablet report...
I20260927 21:35:13.909363 15642 ts_tablet_manager.cc:1428] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 21:35:13.910012 15641 consensus_queue.cc:260] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.913571 15488 heartbeater.cc:503] Master 127.3.186.254:44863 was elected leader, sending a full tablet report...
I20260927 21:35:13.914110 15641 ts_tablet_manager.cc:1428] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Time spent starting tablet: real 0.018s user 0.009s sys 0.005s
I20260927 21:35:13.955754 15646 raft_consensus.cc:482] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:35:13.956027 15646 raft_consensus.cc:504] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.957821 15646 leader_election.cc:288] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30b2282b91b4e91b2cb10186deae4ea (127.3.186.195:38903), 2b7143df8a0942b6bdd4cd7fbc2e6450 (127.3.186.194:39745)
I20260927 21:35:13.966817 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.958730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:38039 (local address 127.3.186.194:39745)
0927 21:35:13.958909 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.958915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.960256 (+ 1341us) server_negotiation.cc:407] Connection header received
0927 21:35:13.960505 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.960510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.960581 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.960667 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:35:13.961154 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.961935 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.964728 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.965917 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.966056 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.966133 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.966448 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.966563 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:35:13.966919 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.958306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:39745 (local address 127.3.186.193:38039)
0927 21:35:13.960165 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.960182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.960338 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.960698 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.960703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.960725 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:35:13.961018 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.961026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.962063 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.962071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.964569 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.964584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.966107 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.966113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.966282 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.966313 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.966399 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.966512 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"mutex_wait_us":143,"thread_start_us":1163,"threads_started":1}
I20260927 21:35:13.967595 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.958411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:43263 (local address 127.3.186.195:38903)
0927 21:35:13.958649 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:13.958658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:13.959325 (+ 667us) server_negotiation.cc:407] Connection header received
0927 21:35:13.959549 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:13.959554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:13.960037 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:13.960109 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:35:13.960883 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.961998 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.965614 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.966563 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.966693 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.966769 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:13.967258 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 21:35:13.967374 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:35:13.967681 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:13.958135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:38903 (local address 127.3.186.193:43263)
0927 21:35:13.959201 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:13.959220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:13.959355 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:13.960324 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:13.960328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:13.960347 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:35:13.960689 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.960699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.962124 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.962128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.964569 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:13.964584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:13.966927 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:13.966936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:13.967100 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:13.967132 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:35:13.967236 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:35:13.967349 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":125,"thread_start_us":717,"threads_started":1}
I20260927 21:35:13.968119 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef96a871a9bf4fcd99f89ca2b9cd87c7" candidate_uuid: "a4ec6910431b40dc81dac9fd70d106c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" is_pre_election: true
I20260927 21:35:13.968637 15443 raft_consensus.cc:2454] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ec6910431b40dc81dac9fd70d106c5 in term 0.
I20260927 21:35:13.968632 15577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef96a871a9bf4fcd99f89ca2b9cd87c7" candidate_uuid: "a4ec6910431b40dc81dac9fd70d106c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b2282b91b4e91b2cb10186deae4ea" is_pre_election: true
I20260927 21:35:13.969159 15577 raft_consensus.cc:2454] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ec6910431b40dc81dac9fd70d106c5 in term 0.
I20260927 21:35:13.969252 15236 leader_election.cc:302] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7143df8a0942b6bdd4cd7fbc2e6450, a4ec6910431b40dc81dac9fd70d106c5; no voters:
I20260927 21:35:13.969682 15646 raft_consensus.cc:2781] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:35:13.969837 15646 raft_consensus.cc:482] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:35:13.969957 15646 raft_consensus.cc:3037] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:35:13.971947 15646 raft_consensus.cc:504] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.972499 15646 leader_election.cc:288] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [CANDIDATE]: Term 1 election: Requested vote from peers c30b2282b91b4e91b2cb10186deae4ea (127.3.186.195:38903), 2b7143df8a0942b6bdd4cd7fbc2e6450 (127.3.186.194:39745)
I20260927 21:35:13.972853 15577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef96a871a9bf4fcd99f89ca2b9cd87c7" candidate_uuid: "a4ec6910431b40dc81dac9fd70d106c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30b2282b91b4e91b2cb10186deae4ea"
I20260927 21:35:13.973057 15577 raft_consensus.cc:3037] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:35:13.973060 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef96a871a9bf4fcd99f89ca2b9cd87c7" candidate_uuid: "a4ec6910431b40dc81dac9fd70d106c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450"
I20260927 21:35:13.973232 15443 raft_consensus.cc:3037] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:35:13.975644 15489 tablet.cc:2369] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:35:13.976476 15577 raft_consensus.cc:2454] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ec6910431b40dc81dac9fd70d106c5 in term 1.
I20260927 21:35:13.976716 15443 raft_consensus.cc:2454] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ec6910431b40dc81dac9fd70d106c5 in term 1.
I20260927 21:35:13.976871 15235 leader_election.cc:302] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ec6910431b40dc81dac9fd70d106c5, c30b2282b91b4e91b2cb10186deae4ea; no voters:
I20260927 21:35:13.977138 15646 raft_consensus.cc:2781] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:35:13.977604 15646 raft_consensus.cc:686] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 1 LEADER]: Becoming Leader. State: Replica: a4ec6910431b40dc81dac9fd70d106c5, State: Running, Role: LEADER
I20260927 21:35:13.978507 15646 consensus_queue.cc:237] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } }
I20260927 21:35:13.983454 15132 catalog_manager.cc:5495] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4ec6910431b40dc81dac9fd70d106c5 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "a4ec6910431b40dc81dac9fd70d106c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ec6910431b40dc81dac9fd70d106c5" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 46113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 } health_report { overall_health: UNKNOWN } } }
W20260927 21:35:13.984383 15623 tablet.cc:2369] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:35:14.003149 15346 tablet.cc:2369] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:35:14.034503 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:14.028999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.3.186.193:46113)
0927 21:35:14.029191 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:14.029199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:14.029216 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:35:14.029433 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:14.029440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:14.029546 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:14.029684 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:35:14.030051 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.031114 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.032456 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.032849 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.032984 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:14.033455 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:35:14.033932 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:35:14.034082 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:14.034184 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:35:14.034256 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:35:14.041805 15577 raft_consensus.cc:1261] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 1 FOLLOWER]: Refusing update from remote peer a4ec6910431b40dc81dac9fd70d106c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:35:14.041800 15443 raft_consensus.cc:1261] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 1 FOLLOWER]: Refusing update from remote peer a4ec6910431b40dc81dac9fd70d106c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:35:14.042605 15651 consensus_queue.cc:1035] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7143df8a0942b6bdd4cd7fbc2e6450" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:35:14.042935 15646 consensus_queue.cc:1035] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30b2282b91b4e91b2cb10186deae4ea" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:35:14.051978 15654 mvcc.cc:205] Tried to move back new op lower bound from 7334071967906508800 to 7334071967660019712. Current Snapshot: MvccSnapshot[applied={T|T < 7334071967906508800}]
I20260927 21:35:14.055828 15656 mvcc.cc:205] Tried to move back new op lower bound from 7334071967906508800 to 7334071967660019712. Current Snapshot: MvccSnapshot[applied={T|T < 7334071967906508800}]
I20260927 21:35:14.064745 15131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:35:14.065032 15131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:35:14.068328 15131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5104369e5234f628e19655c95c715e5: Sending DeleteTablet for 3 replicas of tablet ef96a871a9bf4fcd99f89ca2b9cd87c7
I20260927 21:35:14.069428 15280 tablet_service.cc:1514] Processing DeleteTablet for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:35:14 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:49464
I20260927 21:35:14.069530 15423 tablet_service.cc:1514] Processing DeleteTablet for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:35:14 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:38322
I20260927 21:35:14.070530 15557 tablet_service.cc:1514] Processing DeleteTablet for tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:35:14 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:46386
I20260927 21:35:14.071900 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 15212
I20260927 21:35:14.079867 15663 tablet_replica.cc:330] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: stopping tablet replica
I20260927 21:35:14.080336 15663 raft_consensus.cc:2229] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:35:14.081037 15663 raft_consensus.cc:2258] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:35:14.082702 15663 ts_tablet_manager.cc:1905] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:35:14.089887 15663 ts_tablet_manager.cc:1918] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:35:14.090087 15663 log.cc:1196] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/ef96a871a9bf4fcd99f89ca2b9cd87c7
I20260927 21:35:14.091118 15663 ts_tablet_manager.cc:1939] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P a4ec6910431b40dc81dac9fd70d106c5: Deleting consensus metadata
I20260927 21:35:14.092734 15116 catalog_manager.cc:4841] TS a4ec6910431b40dc81dac9fd70d106c5 (127.3.186.193:46113): tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 (table test-table [id=791dedbcd9684dae9c1291309d3c5108]) successfully deleted
I20260927 21:35:14.094568 15665 tablet_replica.cc:330] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: stopping tablet replica
I20260927 21:35:14.095024 15665 raft_consensus.cc:2229] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:35:14.095495 15665 raft_consensus.cc:2258] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:35:14.095989 15664 tablet_replica.cc:330] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: stopping tablet replica
I20260927 21:35:14.096603 15664 raft_consensus.cc:2229] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:35:14.097064 15664 raft_consensus.cc:2258] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:35:14.098017 15665 ts_tablet_manager.cc:1905] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:35:14.098694 15664 ts_tablet_manager.cc:1905] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:35:14 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:35:14.104543 15665 ts_tablet_manager.cc:1918] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:35:14.104674 15665 log.cc:1196] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/ef96a871a9bf4fcd99f89ca2b9cd87c7
I20260927 21:35:14.105209 15665 ts_tablet_manager.cc:1939] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P c30b2282b91b4e91b2cb10186deae4ea: Deleting consensus metadata
I20260927 21:35:14.105594 15664 ts_tablet_manager.cc:1918] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:35:14.105815 15664 log.cc:1196] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/ef96a871a9bf4fcd99f89ca2b9cd87c7
I20260927 21:35:14.106297 15664 ts_tablet_manager.cc:1939] T ef96a871a9bf4fcd99f89ca2b9cd87c7 P 2b7143df8a0942b6bdd4cd7fbc2e6450: Deleting consensus metadata
I20260927 21:35:14.107501 15118 catalog_manager.cc:4841] TS c30b2282b91b4e91b2cb10186deae4ea (127.3.186.195:38903): tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 (table test-table [id=791dedbcd9684dae9c1291309d3c5108]) successfully deleted
I20260927 21:35:14.107947 15119 catalog_manager.cc:4841] TS 2b7143df8a0942b6bdd4cd7fbc2e6450 (127.3.186.194:39745): tablet ef96a871a9bf4fcd99f89ca2b9cd87c7 (table test-table [id=791dedbcd9684dae9c1291309d3c5108]) successfully deleted
I20260927 21:35:14.111929 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:14.073026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.3.186.193:46113)
0927 21:35:14.077480 (+ 4454us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:14.077488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:14.094009 (+ 16521us) server_negotiation.cc:407] Connection header received
0927 21:35:14.094197 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:14.094205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:14.094301 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:14.094430 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:14.096318 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.097973 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.099765 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.100407 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.100497 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:14.105862 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:14.105874 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:14.105888 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:14.105933 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:14.109183 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:14.109894 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:14.109900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:14.109902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:14.110074 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:14.110419 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:14.110423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:14.110427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:14.110788 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:14.111074 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:14.111386 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 21:35:14.111618 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4301}
I20260927 21:35:14.112598 15340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:14.206295 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 15212
I20260927 21:35:14.223946 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 15353
Sep 27 21:35:14 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:35:14.240511 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:14.224761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.3.186.194:39745)
0927 21:35:14.225086 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:14.225090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:14.225104 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:35:14.225161 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:14.225165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:14.225224 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:14.225294 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:14.226877 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.227749 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.229622 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.230021 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.230145 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:14.235015 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:14.235025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:14.235035 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:14.235062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:14.237913 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:14.238613 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:14.238619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:14.238622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:14.238739 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:14.239129 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:14.239132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:14.239134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:14.239491 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:14.239688 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:14.239986 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:35:14.240132 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260927 21:35:14.241158 15483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:14.325747 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 15353
I20260927 21:35:14.341499 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 15494
Sep 27 21:35:14 dist-test-slave-pqr8 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:35:14.355767 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:14.342197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.3.186.195:38903)
0927 21:35:14.342401 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:14.342416 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:14.342440 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:35:14.342619 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:14.342624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:14.342697 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:14.342799 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:14.344008 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.344716 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.346267 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:14.346655 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:14.346782 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:14.350755 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:14.350768 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:14.350783 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:14.350824 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:14.353405 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:14.354074 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:14.354080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:14.354082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:14.354200 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:14.354504 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:14.354507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:14.354509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:14.354840 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:14.355082 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:14.355294 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 21:35:14.355436 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:35:14.356314 15617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:14.439787 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 15494
I20260927 21:35:14.455237 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 15100
I20260927 21:35:14.456290 15162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:14.578238 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 15100
I20260927 21:35:14.601799 3819 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:35:14.936226 3819 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:35:14.936442 3819 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:35:15.280614 3819 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:35:15.280 UTC [14686] LOG: received smart shutdown request
2026-09-27 21:35:15.283 UTC [14686] LOG: background worker "logical replication launcher" (PID 14695) exited with exit code 1
2026-09-27 21:35:15.283 UTC [14690] LOG: shutting down
2026-09-27 21:35:15.429 UTC [14686] LOG: database system is shut down
2026-09-27T21:35:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46737 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:35:15 dist-test-slave-pqr8 krb5kdc[15679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:35:15 dist-test-slave-pqr8 krb5kdc[15679](info): set up 2 sockets
Sep 27 21:35:15 dist-test-slave-pqr8 krb5kdc[15679](info): commencing operation
krb5kdc: starting...
W20260927 21:35:16.621731 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:35:16 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:35:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/rangeradmin_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/rangeradmin_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/rangerlookup_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/rangerlookup_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
I20260927 21:35:16.888536 3819 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWw8W86/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/postgres -l logfile start
2026-09-27 21:35:19.194 UTC [15709] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:35:19.194 UTC [15709] LOG: listening on IPv4 address "127.3.186.212", port 40863
2026-09-27 21:35:19.198 UTC [15709] LOG: listening on Unix socket "/tmp/.s.PGSQL.40863"
2026-09-27 21:35:19.219 UTC [15712] LOG: database system was shut down at 2026-09-27 21:35:17 UTC
2026-09-27 21:35:19.222 UTC [15709] LOG: database system is ready to accept connections
I20260927 21:35:20.161468 3819 mini_postgres.cc:96] Postgres bound to 40863
2026-09-27 21:35:20.178 UTC [15720] FATAL: database "slave" does not exist
127.3.186.212:40863 - accepting connections
I20260927 21:35:20.178607 3819 mini_ranger.cc:161] Starting Ranger...
I20260927 21:35:20.201665 3819 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:35:20.642086 3819 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:35:20.642176 3819 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-admin
2026-09-27 21:35:20,949 [I] DB FLAVOR :POSTGRES
2026-09-27 21:35:20,950 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:35:20,950 [I] Checking connection..
2026-09-27 21:35:20,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:35:21,172 [I] Checking connection passed.
2026-09-27 21:35:21,172 [I] --------- Verifying version history table ---------
2026-09-27 21:35:21,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:35:21,410 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:35:21,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:35:21,635 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:35:21,635 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:35:21,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:35:21.854 UTC [15784] WARNING: there is no transaction in progress
2026-09-27 21:35:21,859 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:35:21,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:35:22,071 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:35:22,071 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:35:22,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:35:22,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:35:22,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pqr8.c.gcp-upstream.internal','N') ;"
2026-09-27 21:35:22,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:35:22,972 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:35:22,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:35:23,192 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:35:23,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:35:23,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:35:23,647 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:35:23,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:35:24.548 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.566 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.586 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.601 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.615 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.633 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.690 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.696 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.708 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.717 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.727 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.739 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.748 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.753 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.759 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:24.765 UTC [15892] WARNING: there is no transaction in progress
2026-09-27 21:35:25,337 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:35:25,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:35:25,557 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:35:25,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:35:25,800 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:35:25,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:35:26,043 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:35:26,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:35:26,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:35:26,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pqr8.c.gcp-upstream.internal';"
2026-09-27 21:35:26,719 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:35:26,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:35:26,951 [I] DB_PATCHES have already been applied
I20260927 21:35:26.957134 3819 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWw8W86/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:35:26.957242 3819 mini_ranger.cc:193] Using host: 127.3.186.212
I20260927 21:35:26.970758 3819 mini_ranger.cc:227] Ranger admin URL: http://127.3.186.212:36979
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:35:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:35:28.068640 3819 mini_ranger.cc:160] Time spent starting Ranger: real 7.890s user 0.004s sys 0.029s
Sep 27, 2026 9:35:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:35:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 9:35:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:35:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:35:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:35:48.312106 3819 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/rangerkms_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/rangerkms_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.186.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.186.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.186.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/HTTP_127.3.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:35:48 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544948, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:35:48.456789 3819 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:35:48.473160 3819 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:35:49.261494 3819 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:35:49.261636 3819 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms
I20260927 21:35:49.261680 3819 mini_ranger_kms.cc:227] Using postgres at 127.3.186.212:40863
2026-09-27 21:35:49,606 [I] DB FLAVOR :POSTGRES
2026-09-27 21:35:49,606 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:35:49,606 [I] Checking connection
2026-09-27 21:35:49,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:35:49,851 [I] connection success
2026-09-27 21:35:49,851 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:35:49,851 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:35:49,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:35:50,067 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:35:50,067 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:35:50,067 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:35:50,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.186.212:40863/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:35:50,304 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:35:50.709143 3819 mini_ranger_kms.cc:310] Created kms service
I20260927 21:35:50.901109 3819 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:35:50.941752 3819 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:35:50 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544948, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:51.305032 3819 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:35:51.305371 3819 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWw8W86/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWw8W86/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:35:51.305441 3819 mini_ranger_kms.cc:240] Using host: 127.3.186.212
I20260927 21:35:51.319330 3819 mini_ranger_kms.cc:276] Ranger KMS PID: 16072
I20260927 21:35:51.319437 3819 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.186.212:42035
Sep 27, 2026 9:35:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:35:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:35:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:35:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:35:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:35:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 21:35:53 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544953, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.186.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:53.499930 3819 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.043s user 0.048s sys 0.306s
I20260927 21:35:53.500183 3819 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:35:53.500352 3819 mini_ranger_kms.cc:401] 127.3.186.212:42035/kms/v1/keys
Sep 27 21:35:53 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544953, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.186.212@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
Sep 27 21:35:54 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544953, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.186.212@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
Sep 27 21:35:56 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:35:56.260610 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40421
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:40421
--ranger_config_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:35:56.567577 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:56.567826 16123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:56.567989 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:56.575778 16123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:35:56.575865 16123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:56.575960 16123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:56.576040 16123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:35:56.576104 16123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:35:56.585666 16123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.186.212:42035/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:40421
--ranger_config_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:56.586743 16123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:56.587966 16123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:56.599165 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:56.599255 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:56.599255 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:56.599872 16123 server_base.cc:1034] running on GCE node
I20260927 21:35:56.600579 16123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:56.601799 16123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:56.603024 16123 hybrid_clock.cc:648] HybridClock initialized: now 1790544956602979 us; error 41 us; skew 500 ppm
I20260927 21:35:56.603309 16123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:56 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:56.606222 16123 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:35:56.607556 16123 webserver.cc:460] Webserver started at http://127.3.186.254:39677/ using document root <none> and password file <none>
I20260927 21:35:56.608103 16123 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:56.608191 16123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:56.608418 16123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:56 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:56.665843 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "f932d6d97b684c11a882a1eb0fd00240"
format_stamp: "Formatted at 2026-09-27 21:35:56 on dist-test-slave-pqr8"
server_key: "deeddf5d999e5bb77f012980ddf53378"
server_key_iv: "67a5808a5e214a2ee3bcdb5af8dead49"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:56.666769 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "f932d6d97b684c11a882a1eb0fd00240"
format_stamp: "Formatted at 2026-09-27 21:35:56 on dist-test-slave-pqr8"
server_key: "deeddf5d999e5bb77f012980ddf53378"
server_key_iv: "67a5808a5e214a2ee3bcdb5af8dead49"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:56.673240 16123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 21:35:56.690663 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:56.692227 16123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:35:56.692406 16123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "f932d6d97b684c11a882a1eb0fd00240"
format_stamp: "Formatted at 2026-09-27 21:35:56 on dist-test-slave-pqr8"
server_key: "deeddf5d999e5bb77f012980ddf53378"
server_key_iv: "67a5808a5e214a2ee3bcdb5af8dead49"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:56.692633 16123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:56.735471 16123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:56.739904 16123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:56.740209 16123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:56.759346 16189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:40421 every 8 connection(s)
I20260927 21:35:56.759373 16123 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:40421
I20260927 21:35:56.760957 16123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:35:56.764482 16190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:35:56.770982 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16123
I20260927 21:35:56.771204 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:35:56.771991 3819 external_mini_cluster.cc:1428] Setting key f4c7f577b3b4719d552b03aaf7df1952
I20260927 21:35:56.775494 16190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Bootstrap starting.
I20260927 21:35:56.780256 16190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Neither blocks nor log segments found. Creating new log.
I20260927 21:35:56.781899 16190 log.cc:824] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Log is configured to *not* fsync() on all Append() calls
I20260927 21:35:56.791042 16190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: No bootstrap required, opened a new log
I20260927 21:35:56.800356 16190 raft_consensus.cc:348] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:35:56.800698 16190 raft_consensus.cc:374] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:35:56.800815 16190 raft_consensus.cc:729] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f932d6d97b684c11a882a1eb0fd00240, State: Initialized, Role: FOLLOWER
I20260927 21:35:56.801574 16190 consensus_queue.cc:260] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:35:56.801859 16190 raft_consensus.cc:388] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 27 21:35:56 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:56.802014 16190 raft_consensus.cc:482] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:35:56.802194 16190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:35:56.805122 16190 raft_consensus.cc:504] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:35:56.805763 16190 leader_election.cc:302] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f932d6d97b684c11a882a1eb0fd00240; no voters:
I20260927 21:35:56.806408 16190 leader_election.cc:288] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:35:56.806522 16195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:35:56.813803 16193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:56.773995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.3.186.254:40421)
0927 21:35:56.786159 (+ 12164us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:56.786180 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:56.788241 (+ 2061us) server_negotiation.cc:407] Connection header received
0927 21:35:56.790800 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:56.790827 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:56.791287 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:56.791700 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:56.793274 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:56.795402 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:56.797268 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:56.798359 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:56.798526 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:56.803199 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:56.803215 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:56.803233 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:56.803272 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:56.807295 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:56.808403 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:56.808414 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:56.808426 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:56.808624 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:56.809136 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:56.809143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:56.809149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:56.810004 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:56.811688 (+ 1684us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:56.812007 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:35:56.812480 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11861,"thread_start_us":11588,"threads_started":1}
I20260927 21:35:56.814819 16190 sys_catalog.cc:564] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:35:56.815047 16195 raft_consensus.cc:686] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 LEADER]: Becoming Leader. State: Replica: f932d6d97b684c11a882a1eb0fd00240, State: Running, Role: LEADER
I20260927 21:35:56.815681 16195 consensus_queue.cc:237] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:35:56.819005 16196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f932d6d97b684c11a882a1eb0fd00240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } } }
I20260927 21:35:56.820001 16196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: This master's current role is: LEADER
I20260927 21:35:56.819658 16197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f932d6d97b684c11a882a1eb0fd00240. Latest consensus state: current_term: 1 leader_uuid: "f932d6d97b684c11a882a1eb0fd00240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } } }
I20260927 21:35:56.820518 16197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: This master's current role is: LEADER
I20260927 21:35:56.824975 16201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:35:56.830857 16201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:35:56.851966 16201 catalog_manager.cc:1347] Generated new cluster ID: 161bc7d9efe3476a8b0cef2674723bba
I20260927 21:35:56.852068 16201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:35:56.865365 16201 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:35:56.867092 16201 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:35:56.874907 16190 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:35:56.888999 16201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Generated new TSK 0
I20260927 21:35:56.889969 16201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:35:57.616910 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40421
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:35:57 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:35:58.028272 16235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:58.028668 16235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:58.028882 16235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:58.040077 16235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:58.040248 16235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:58.040462 16235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:35:58.055261 16235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.186.212:42035/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:58.056977 16235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:58.058591 16235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:58.076651 16245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:58.077913 16246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:58.085080 16248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:58.086067 16235 server_base.cc:1034] running on GCE node
I20260927 21:35:58.086843 16235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:58.088130 16235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:58.089468 16235 hybrid_clock.cc:648] HybridClock initialized: now 1790544958089365 us; error 83 us; skew 500 ppm
I20260927 21:35:58.089932 16235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:58.093936 16235 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:35:58.113123 16235 webserver.cc:460] Webserver started at http://127.3.186.193:37829/ using document root <none> and password file <none>
I20260927 21:35:58.113869 16235 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:58.114006 16235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:58.114358 16235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:58.132665 16235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "9a9b2fd1117c43c9aedf9f3f92409e14"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "41602f803a01f977a8f685cd79ec3174"
server_key_iv: "d2ae71c2bf72ab8d54018929de191906"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:58.133747 16235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "9a9b2fd1117c43c9aedf9f3f92409e14"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "41602f803a01f977a8f685cd79ec3174"
server_key_iv: "d2ae71c2bf72ab8d54018929de191906"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:58.140069 16235 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:35:58.179785 16255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:58.181355 16235 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260927 21:35:58.181593 16235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "9a9b2fd1117c43c9aedf9f3f92409e14"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "41602f803a01f977a8f685cd79ec3174"
server_key_iv: "d2ae71c2bf72ab8d54018929de191906"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:58.181942 16235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:58.227821 16235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:58.232532 16235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:58.232951 16235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:58.249476 16235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:58.262265 16235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:35:58.262404 16235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:58.262547 16235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:35:58.262629 16235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:58.353087 16235 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:41851
I20260927 21:35:58.353127 16367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:41851 every 8 connection(s)
I20260927 21:35:58.354735 16235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:35:58.363240 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16235
I20260927 21:35:58.363462 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:35:58.364468 3819 external_mini_cluster.cc:1428] Setting key 6b4a05aa102bd35d82dcafe753c61b5e
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:58.392259 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:58.358545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:37805 (local address 127.3.186.254:40421)
0927 21:35:58.369355 (+ 10810us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:58.369362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:58.369379 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:35:58.369429 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:58.369436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:58.369537 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:58.369631 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:58.376489 (+ 6858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:58.377597 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:58.380012 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:58.380732 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:58.380843 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:58.385711 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:58.385717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:58.385722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:58.385756 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:58.388928 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:58.389847 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:58.389855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:58.389861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:58.390013 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:58.390581 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:58.390588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:58.390592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:58.390916 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:58.391016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:58.391703 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 21:35:58.391893 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10674,"thread_start_us":15566,"threads_started":1}
I20260927 21:35:58.393424 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:58.357968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40421 (local address 127.3.186.193:37805)
0927 21:35:58.366049 (+ 8081us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:58.366092 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:58.368653 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:58.374878 (+ 6225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:58.374890 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:58.375504 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:35:58.376295 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:58.376319 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:58.377785 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:58.377790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:58.379837 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:58.379855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:58.381206 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:58.381214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:58.381598 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:58.382465 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:35:58.382498 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:35:58.385402 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:35:58.389061 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:58.389071 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:58.389089 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:58.389699 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:58.390178 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:58.390186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:58.390192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:58.390449 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:58.391084 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:35:58.391090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:35:58.391358 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 21:35:58.391598 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 21:35:58.392054 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7827,"thread_start_us":7596,"threads_started":1}
I20260927 21:35:58.395155 16368 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40421
I20260927 21:35:58.395469 16368 heartbeater.cc:464] Registering TS with master...
I20260927 21:35:58.396127 16368 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:35:58.398766 16155 ts_manager.cc:194] Registered new tserver with Master: 9a9b2fd1117c43c9aedf9f3f92409e14 (127.3.186.193:41851)
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
I20260927 21:35:58.400640 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:37805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:35:58.430464 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40421
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
W20260927 21:35:58.848578 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:58.848829 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:58.848917 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:58.855536 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:58.855620 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:58.855720 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:35:58.864351 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.186.212:42035/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:58.865427 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:58.866605 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:35:58.878646 16391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:58.878607 16389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:58.878681 16388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:58.879441 16374 server_base.cc:1034] running on GCE node
I20260927 21:35:58.880007 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:58.880805 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:58.882022 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790544958881993 us; error 44 us; skew 500 ppm
I20260927 21:35:58.882299 16374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:58.885119 16374 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:35:58.886596 16374 webserver.cc:460] Webserver started at http://127.3.186.194:39683/ using document root <none> and password file <none>
I20260927 21:35:58.887321 16374 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:58.887454 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:58.887723 16374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:58 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:58.902696 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "bd662b5d73864a66ae5db2229442ee80"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "de0ed6941ef3827964c45290f640cadb"
server_key_iv: "a70c62f4bf4f8249005e5da7ce7bc419"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:58.903668 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "bd662b5d73864a66ae5db2229442ee80"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "de0ed6941ef3827964c45290f640cadb"
server_key_iv: "a70c62f4bf4f8249005e5da7ce7bc419"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:58.909693 16374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 21:35:58.921970 16398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:58.923290 16374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:35:58.923476 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "bd662b5d73864a66ae5db2229442ee80"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "de0ed6941ef3827964c45290f640cadb"
server_key_iv: "a70c62f4bf4f8249005e5da7ce7bc419"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:58.923683 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:58.947402 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:58.951215 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:58.951481 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:58.952303 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:35:58.953734 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:35:58.953828 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:58.953966 16374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:35:58.954031 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:58.992004 16374 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:42147
I20260927 21:35:58.992022 16510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:42147 every 8 connection(s)
I20260927 21:35:58.993216 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:35:59.000008 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16374
I20260927 21:35:59.000162 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:35:59.000715 3819 external_mini_cluster.cc:1428] Setting key f424fcbe34d9a8534eee78badc6ae0f1
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544958, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:59.016122 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:58.996160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:34023 (local address 127.3.186.254:40421)
0927 21:35:58.996876 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:58.996882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:58.997711 (+ 829us) server_negotiation.cc:407] Connection header received
0927 21:35:58.999533 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:58.999539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:58.999632 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:58.999760 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:59.001583 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.002667 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.004994 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.005766 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.005911 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.010169 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:59.010174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:59.010177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:59.010208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:59.012941 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.013690 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.013694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.013696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.013817 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.014243 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.014252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.014255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.014535 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:59.014696 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:59.015438 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 21:35:59.015661 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":442,"threads_started":1}
I20260927 21:35:59.017046 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:58.996792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40421 (local address 127.3.186.194:34023)
0927 21:35:58.997558 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:58.997601 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:58.999276 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:59.000018 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:59.000031 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:59.000618 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:35:59.001389 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:59.001409 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.002840 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.002846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:59.004842 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:59.004857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.006001 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.006008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:59.006293 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.007085 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:35:59.007112 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:35:59.009955 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:35:59.013092 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:59.013101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:59.013115 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:59.013567 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:59.013926 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:59.013930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:59.013934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:59.014122 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:59.014721 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:35:59.014729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:35:59.015038 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 21:35:59.015346 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 21:35:59.015923 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":323,"threads_started":1}
I20260927 21:35:59.018541 16511 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40421
I20260927 21:35:59.018868 16511 heartbeater.cc:464] Registering TS with master...
I20260927 21:35:59.019663 16511 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:35:59.021417 16155 ts_manager.cc:194] Registered new tserver with Master: bd662b5d73864a66ae5db2229442ee80 (127.3.186.194:42147)
I20260927 21:35:59.022819 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:34023
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:35:59.060595 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40421
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:35:59.380080 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:35:59.380312 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:35:59.380462 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:35:59.387565 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:35:59.387655 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:35:59.387846 16517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:35:59.396608 16517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.186.212:42035/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:35:59.397735 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:35:59.398944 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:35:59.403620 16368 heartbeater.cc:503] Master 127.3.186.254:40421 was elected leader, sending a full tablet report...
W20260927 21:35:59.411413 16522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:59.411386 16523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:35:59.411437 16525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:35:59.412473 16517 server_base.cc:1034] running on GCE node
I20260927 21:35:59.412986 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:35:59.413844 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:35:59.415086 16517 hybrid_clock.cc:648] HybridClock initialized: now 1790544959415047 us; error 43 us; skew 500 ppm
I20260927 21:35:59.415402 16517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:35:59.418319 16517 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:35:59.419548 16517 webserver.cc:460] Webserver started at http://127.3.186.195:36433/ using document root <none> and password file <none>
I20260927 21:35:59.420140 16517 fs_manager.cc:359] Metadata directory not provided
I20260927 21:35:59.420277 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:35:59.420550 16517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:35:59.437228 16517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "6f46b6e714134d4db2979b4626d92f09"
format_stamp: "Formatted at 2026-09-27 21:35:59 on dist-test-slave-pqr8"
server_key: "c21703496518f41abaa9b9fee922d93b"
server_key_iv: "496f73c8826b2551791b8147c4e3633c"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:59.438256 16517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "6f46b6e714134d4db2979b4626d92f09"
format_stamp: "Formatted at 2026-09-27 21:35:59 on dist-test-slave-pqr8"
server_key: "c21703496518f41abaa9b9fee922d93b"
server_key_iv: "496f73c8826b2551791b8147c4e3633c"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:59.444350 16517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:35:59.458410 16532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:59.459746 16517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:35:59.459908 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "6f46b6e714134d4db2979b4626d92f09"
format_stamp: "Formatted at 2026-09-27 21:35:59 on dist-test-slave-pqr8"
server_key: "c21703496518f41abaa9b9fee922d93b"
server_key_iv: "496f73c8826b2551791b8147c4e3633c"
server_key_version: "kuduclusterkey@0"
I20260927 21:35:59.460098 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:35:59.477478 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:35:59.480657 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:35:59.480949 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:35:59.481837 16517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:35:59.483022 16517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:35:59.483090 16517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:59.483227 16517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:35:59.483284 16517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:35:59.514533 16517 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:40389
I20260927 21:35:59.514572 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:40389 every 8 connection(s)
I20260927 21:35:59.515781 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:35:59.521915 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16517
I20260927 21:35:59.522078 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:35:59.522675 3819 external_mini_cluster.cc:1428] Setting key e83d29634f32de30908393d4c308f311
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544959, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:35:59.534094 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:59.518242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:33281 (local address 127.3.186.254:40421)
0927 21:35:59.518664 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:59.518669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:59.519474 (+ 805us) server_negotiation.cc:407] Connection header received
0927 21:35:59.521151 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:59.521157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:59.521239 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:59.521362 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:59.523195 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.523921 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.525541 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.525941 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.526081 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.529257 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:59.529262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:59.529266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:59.529309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:59.531716 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.532323 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.532326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.532328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.532446 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.532793 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.532796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.532798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.533031 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:59.533134 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:59.533701 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 21:35:59.533855 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260927 21:35:59.534929 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:59.518703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40421 (local address 127.3.186.195:33281)
0927 21:35:59.519324 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 21:35:59.519363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:35:59.520904 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:35:59.521662 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:35:59.521674 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:35:59.522234 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:35:59.522982 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:59.523008 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.524086 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.524091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:59.525383 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:35:59.525394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.526155 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.526162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:35:59.526384 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.527044 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:35:59.527065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:35:59.529075 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:35:59.531846 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:59.531851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:59.531862 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:59.532220 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:59.532536 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:35:59.532539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:35:59.532541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:35:59.532706 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:35:59.533130 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:35:59.533135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:35:59.533379 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 21:35:59.533631 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 21:35:59.533933 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260927 21:35:59.536121 16645 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40421
I20260927 21:35:59.536397 16645 heartbeater.cc:464] Registering TS with master...
I20260927 21:35:59.536988 16645 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:35:59.538288 16155 ts_manager.cc:194] Registered new tserver with Master: 6f46b6e714134d4db2979b4626d92f09 (127.3.186.195:40389)
I20260927 21:35:59.538980 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:33281
I20260927 21:35:59.549256 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:35:59.550926 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16235
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:35:59.566896 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:59.551789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.3.186.193:41851)
0927 21:35:59.552381 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:59.552386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:59.552408 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:35:59.552456 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:59.552461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:59.552610 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:59.552716 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:59.554249 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.555468 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.556927 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.557433 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.557607 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.561263 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:59.561273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:59.561283 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:59.561316 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:59.564287 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.565022 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.565028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.565030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.565166 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.565483 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.565486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.565488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.565823 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:59.566103 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:59.566349 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 21:35:59.566534 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":318,"threads_started":1}
I20260927 21:35:59.567953 16363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:59.648334 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16235
I20260927 21:35:59.663708 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16374
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:35:59.679267 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:59.664621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.3.186.194:42147)
0927 21:35:59.665181 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:59.665188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:59.665213 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:35:59.665292 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:59.665298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:59.665504 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:59.665717 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:59.666907 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.667928 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.669179 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.669870 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.669972 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.674187 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:59.674198 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:59.674210 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:59.674240 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:59.676648 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.677289 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.677298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.677301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.677425 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.677818 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.677823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.677826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.678128 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:59.678411 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:59.678685 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:35:59.678943 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
I20260927 21:35:59.680456 16506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:59.760021 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16374
I20260927 21:35:59.779402 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16517
Sep 27 21:35:59 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:35:59.802920 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:35:59.780221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.3.186.195:40389)
0927 21:35:59.780816 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 21:35:59.780823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:35:59.780849 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:35:59.780903 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:35:59.780926 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:35:59.781128 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:35:59.781248 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:35:59.782917 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.784679 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.786604 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:35:59.787614 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:35:59.787762 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:35:59.797824 (+ 10062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:35:59.797839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:35:59.797853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:35:59.797886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:35:59.800695 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.801317 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.801323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.801325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.801441 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:35:59.801792 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:35:59.801795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:35:59.801797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:35:59.802114 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:35:59.802334 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 21:35:59.802512 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:35:59.802675 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260927 21:35:59.804142 16640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:35:59.882438 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16517
I20260927 21:35:59.897403 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16123
I20260927 21:35:59.898552 16185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:00.000465 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16123
I20260927 21:36:00.020848 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40421
--webserver_interface=127.3.186.254
--webserver_port=39677
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:40421
--ranger_config_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:00.321435 16657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:00.321739 16657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:00.321885 16657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:00.328701 16657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:00.328821 16657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:00.328919 16657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:00.329011 16657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:00.329077 16657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:00.338050 16657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:40421
--ranger_config_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:00.339093 16657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:00.340327 16657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:00.351493 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:00.351679 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:00.351773 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:00.352445 16657 server_base.cc:1034] running on GCE node
I20260927 21:36:00.353072 16657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:00.354346 16657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:00.355536 16657 hybrid_clock.cc:648] HybridClock initialized: now 1790544960355485 us; error 52 us; skew 500 ppm
I20260927 21:36:00.355892 16657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:00 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:00.358982 16657 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:00.360288 16657 webserver.cc:460] Webserver started at http://127.3.186.254:39677/ using document root <none> and password file <none>
I20260927 21:36:00.360872 16657 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:00.361004 16657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:36:00 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:36:00.429965 16657 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:36:00.432971 16673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:00.434381 16657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:36:00.434566 16657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "f932d6d97b684c11a882a1eb0fd00240"
format_stamp: "Formatted at 2026-09-27 21:35:56 on dist-test-slave-pqr8"
server_key: "deeddf5d999e5bb77f012980ddf53378"
server_key_iv: "67a5808a5e214a2ee3bcdb5af8dead49"
server_key_version: "kuduclusterkey@0"
I20260927 21:36:00.435492 16657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:00.447965 16657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:00.451326 16657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:00.451620 16657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:00.468750 16657 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:40421
I20260927 21:36:00.468792 16724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:40421 every 8 connection(s)
I20260927 21:36:00.470120 16657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:00.476275 16725 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:36:00.479028 16725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Bootstrap starting.
I20260927 21:36:00.480232 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16657
I20260927 21:36:00.481364 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:41851
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:40421
--webserver_port=37829
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:00.498972 16725 log.cc:824] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:00.506791 16725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:36:00.507311 16725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Bootstrap complete.
I20260927 21:36:00.515683 16725 raft_consensus.cc:348] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:36:00.516418 16725 raft_consensus.cc:729] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f932d6d97b684c11a882a1eb0fd00240, State: Initialized, Role: FOLLOWER
I20260927 21:36:00.516976 16725 consensus_queue.cc:260] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:36:00.517180 16725 raft_consensus.cc:388] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:00.517318 16725 raft_consensus.cc:482] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:00.517499 16725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:36:00.520895 16725 raft_consensus.cc:504] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:36:00.521348 16725 leader_election.cc:302] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f932d6d97b684c11a882a1eb0fd00240; no voters:
I20260927 21:36:00.521958 16725 leader_election.cc:288] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:36:00.522086 16729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:36:00.523113 16729 raft_consensus.cc:686] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [term 2 LEADER]: Becoming Leader. State: Replica: f932d6d97b684c11a882a1eb0fd00240, State: Running, Role: LEADER
I20260927 21:36:00.523648 16729 consensus_queue.cc:237] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } }
I20260927 21:36:00.523940 16725 sys_catalog.cc:564] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:00.525995 16731 sys_catalog.cc:455] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f932d6d97b684c11a882a1eb0fd00240. Latest consensus state: current_term: 2 leader_uuid: "f932d6d97b684c11a882a1eb0fd00240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } } }
I20260927 21:36:00.526326 16731 sys_catalog.cc:458] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:00.526856 16730 sys_catalog.cc:455] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f932d6d97b684c11a882a1eb0fd00240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f932d6d97b684c11a882a1eb0fd00240" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40421 } } }
I20260927 21:36:00.527118 16730 sys_catalog.cc:458] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:00.531388 16735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:00.538070 16735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:00.539785 16735 catalog_manager.cc:1259] Loaded cluster ID: 161bc7d9efe3476a8b0cef2674723bba
I20260927 21:36:00.539989 16735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:00.544627 16735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:00.546409 16735 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Loaded TSK: 0
I20260927 21:36:00.547994 16725 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:36:00.548614 16735 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:36:00.862376 16727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:00.862761 16727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:00.862998 16727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:00.872978 16727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:00.873143 16727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:00.873384 16727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:36:00.887953 16727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:41851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=37829
--tserver_master_addrs=127.3.186.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:00.889103 16727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:00.890421 16727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:00.923566 16763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:00.926039 16762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:00.927791 16765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:00.928371 16727 server_base.cc:1034] running on GCE node
I20260927 21:36:00.929219 16727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:00.937822 16727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:00.939148 16727 hybrid_clock.cc:648] HybridClock initialized: now 1790544960939041 us; error 90 us; skew 500 ppm
I20260927 21:36:00.939563 16727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:00 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:00.943542 16727 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:36:00.957331 16727 webserver.cc:460] Webserver started at http://127.3.186.193:37829/ using document root <none> and password file <none>
I20260927 21:36:00.958120 16727 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:00.958271 16727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:36:00 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:36:00.979060 16727 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:36:00.981786 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:00.983413 16727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:36:00.983582 16727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "9a9b2fd1117c43c9aedf9f3f92409e14"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "41602f803a01f977a8f685cd79ec3174"
server_key_iv: "d2ae71c2bf72ab8d54018929de191906"
server_key_version: "kuduclusterkey@0"
I20260927 21:36:00.984517 16727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:01.015204 16727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:01.020061 16727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:01.020354 16727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:01.037568 16727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:01.039109 16727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:01.039229 16727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:01.039371 16727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:01.039462 16727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:01.177807 16727 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:41851
I20260927 21:36:01.179572 16727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:36:01.182132 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16727
I20260927 21:36:01.182893 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:42147
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:40421
--webserver_port=39683
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:01.187008 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:41851 every 8 connection(s)
Sep 27 21:36:01 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:01.225354 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:01.189770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40421 (local address 127.3.186.193:60245)
0927 21:36:01.190622 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:01.190671 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:01.197498 (+ 6827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:01.202895 (+ 5397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:01.202907 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:01.203588 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:01.204395 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:01.204422 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.206993 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.206999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:01.209292 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:01.209318 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.210860 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.210866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:01.211109 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:01.213715 (+ 2606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:01.213745 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:01.216727 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:01.220884 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:01.220892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:01.220904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:01.221447 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:01.221950 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:01.221955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:01.221958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:01.222180 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:01.223160 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:01.223167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:01.223514 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 21:36:01.223778 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 21:36:01.224207 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":1282,"threads_started":1}
I20260927 21:36:01.226109 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:01.190895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:60245 (local address 127.3.186.254:40421)
0927 21:36:01.200793 (+ 9898us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:01.200812 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:01.200862 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:36:01.201272 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:01.201310 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:01.201826 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:01.202220 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:01.204650 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.206780 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.209458 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.210408 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.210556 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:01.217002 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:01.217011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:01.217028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:01.217071 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:01.220719 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:01.221625 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:01.221635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:01.221642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:01.221820 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:01.222346 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:01.222351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:01.222354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:01.222978 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:01.223175 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:01.224030 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 21:36:01.224445 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9620,"thread_start_us":9442,"threads_started":1}
I20260927 21:36:01.229243 16887 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40421
I20260927 21:36:01.229586 16887 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:01.230347 16887 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
W20260927 21:36:01.233163 16887 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:40421 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:36:01.233462 16887 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:01.233719 16887 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:36:01.234910 16887 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:01.235184 16887 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
W20260927 21:36:01.236196 16887 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:36:01.541887 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:01.542163 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:01.542340 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:01.549788 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:01.549919 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:01.550067 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:36:01.559914 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:42147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=39683
--tserver_master_addrs=127.3.186.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:01.561013 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:01.562309 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:01.576399 16906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:01.576517 16904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:01.577008 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:01.578166 16889 server_base.cc:1034] running on GCE node
I20260927 21:36:01.578824 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:01.581355 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:01.582597 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790544961582454 us; error 127 us; skew 500 ppm
I20260927 21:36:01.582969 16889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:01 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:01.587584 16889 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:36:01.597880 16889 webserver.cc:460] Webserver started at http://127.3.186.194:39683/ using document root <none> and password file <none>
I20260927 21:36:01.598701 16889 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:01.598842 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:36:01 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:36:01.625046 16889 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.009s sys 0.000s
I20260927 21:36:01.628481 16916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:01.630290 16889 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 21:36:01.630461 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "bd662b5d73864a66ae5db2229442ee80"
format_stamp: "Formatted at 2026-09-27 21:35:58 on dist-test-slave-pqr8"
server_key: "de0ed6941ef3827964c45290f640cadb"
server_key_iv: "a70c62f4bf4f8249005e5da7ce7bc419"
server_key_version: "kuduclusterkey@0"
I20260927 21:36:01.631400 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:01.679273 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:01.684000 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:01.684283 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:01.691506 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:01.699208 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:01.699323 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:01.699421 16889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:01.699497 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:01.748344 16889 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:42147
I20260927 21:36:01.748524 17030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:42147 every 8 connection(s)
I20260927 21:36:01.750131 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:36:01 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:01.759464 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 16889
I20260927 21:36:01.760073 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.186.212:42035/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:40389
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:40421
--webserver_port=36433
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:36:01 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:01.775293 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:01.752810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:60469 (local address 127.3.186.254:40421)
0927 21:36:01.753336 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:01.753342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:01.754500 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 21:36:01.757064 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:01.757070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:01.757148 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:01.757231 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:01.759212 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.760405 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.762774 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.763437 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.763542 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:01.768015 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:01.768022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:01.768027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:01.768067 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:01.771633 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:01.772542 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:01.772548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:01.772552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:01.772698 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:01.773250 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:01.773256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:01.773259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:01.773623 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:01.773752 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:01.774691 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 21:36:01.774910 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
I20260927 21:36:01.777773 17034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:01.753478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40421 (local address 127.3.186.194:60469)
0927 21:36:01.754317 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:01.754374 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:01.756769 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:01.757740 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:01.757751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:01.758284 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:01.759019 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:01.759042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.760599 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.760604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:01.762596 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:01.762629 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:01.763888 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:01.763895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:01.764250 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:01.765009 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:01.765037 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:01.767771 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:01.771791 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:01.771802 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:01.771823 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:01.772413 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:01.772835 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:01.772840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:01.772843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:01.773112 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:01.773843 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:01.773850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:01.774237 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 21:36:01.775796 (+ 1559us) client_negotiation.cc:240] Negotiation successful
0927 21:36:01.776378 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":324,"threads_started":1}
I20260927 21:36:01.779675 17031 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40421
I20260927 21:36:01.780001 17031 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:01.780830 17031 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:36:01.788462 16690 ts_manager.cc:194] Registered new tserver with Master: bd662b5d73864a66ae5db2229442ee80 (127.3.186.194:42147)
I20260927 21:36:01.791025 16690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:60469
Sep 27 21:36:02 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
W20260927 21:36:02.229456 17035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:02.229854 17035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:02.230044 17035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:36:02.236363 16887 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:02.236786 16887 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:36:02.238003 16690 ts_manager.cc:194] Registered new tserver with Master: 9a9b2fd1117c43c9aedf9f3f92409e14 (127.3.186.193:41851)
I20260927 21:36:02.239169 16690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:60245
W20260927 21:36:02.240300 17035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:02.240404 17035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:02.240553 17035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:36:02.252835 17035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.186.212:42035/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:40389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=36433
--tserver_master_addrs=127.3.186.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:02.254120 17035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:02.255477 17035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:02.269183 17041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:02.269148 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:02.269762 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:02.270140 17035 server_base.cc:1034] running on GCE node
I20260927 21:36:02.270715 17035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:02.271677 17035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:02.272888 17035 hybrid_clock.cc:648] HybridClock initialized: now 1790544962272838 us; error 42 us; skew 500 ppm
I20260927 21:36:02.273181 17035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:02 dist-test-slave-pqr8 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544962, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:02.276568 17035 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:36:02.278385 17035 webserver.cc:460] Webserver started at http://127.3.186.195:36433/ using document root <none> and password file <none>
I20260927 21:36:02.279213 17035 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:02.279353 17035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:36:02 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544962, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:36:02.301890 17035 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:36:02.305182 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:02.306816 17035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:36:02.307025 17035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "6f46b6e714134d4db2979b4626d92f09"
format_stamp: "Formatted at 2026-09-27 21:35:59 on dist-test-slave-pqr8"
server_key: "c21703496518f41abaa9b9fee922d93b"
server_key_iv: "496f73c8826b2551791b8147c4e3633c"
server_key_version: "kuduclusterkey@0"
I20260927 21:36:02.308382 17035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:02.335327 17035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:02.340102 17035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:02.340428 17035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:02.341699 17035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:02.343381 17035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:02.343528 17035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:36:02.343683 17035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:02.343802 17035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:02.393640 17035 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:40389
I20260927 21:36:02.393682 17164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:40389 every 8 connection(s)
I20260927 21:36:02.395196 17035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:36:02.402153 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17035
Sep 27 21:36:02 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544962, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:02.433467 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.399899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:44529 (local address 127.3.186.254:40421)
0927 21:36:02.401022 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.401028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.411937 (+ 10909us) server_negotiation.cc:407] Connection header received
0927 21:36:02.412014 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.412020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.412114 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.412223 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:02.415609 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.416778 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.419909 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.420547 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.420696 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.426137 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:02.426145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:02.426150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:02.426195 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:02.429861 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.430789 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.430795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.430799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.430934 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.431431 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.431436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.431439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.431825 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:02.431961 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.432836 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.433064 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":830,"threads_started":1}
I20260927 21:36:02.434854 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.398949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40421 (local address 127.3.186.195:44529)
0927 21:36:02.401440 (+ 2491us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:02.401489 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:02.410076 (+ 8587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:02.412660 (+ 2584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:02.412671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:02.414046 (+ 1375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:02.415399 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.415424 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.417743 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.417752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.419747 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.419761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.421023 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.421032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.421294 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.422470 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:02.422511 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:02.425904 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:02.430037 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:02.430046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:02.430061 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:02.430634 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:02.431058 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:02.431064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:02.431068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:02.431298 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:02.432028 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:02.432037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:02.432406 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 21:36:02.432687 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 21:36:02.433329 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2184,"thread_start_us":1931,"threads_started":1}
I20260927 21:36:02.436769 17165 heartbeater.cc:347] Connected to a master server at 127.3.186.254:40421
I20260927 21:36:02.437072 17165 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:02.437955 17165 heartbeater.cc:511] Master 127.3.186.254:40421 requested a full tablet report, sending...
I20260927 21:36:02.439675 16690 ts_manager.cc:194] Registered new tserver with Master: 6f46b6e714134d4db2979b4626d92f09 (127.3.186.195:40389)
I20260927 21:36:02.440884 16690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:44529
I20260927 21:36:02.448083 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.403174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.3.186.254:40421)
0927 21:36:02.421362 (+ 18188us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.421368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.421385 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:36:02.421430 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.421435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.421535 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.422103 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:02.424274 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.425597 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.437580 (+ 11983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.438342 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.438485 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.441929 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:02.441936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:02.441940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:02.441976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:02.445076 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.445898 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.445904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.445907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.446055 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.446485 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.446490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.446494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.446814 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:02.447057 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.447472 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.447691 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18062,"thread_start_us":18773,"threads_started":1}
I20260927 21:36:02.450755 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:36:02 dist-test-slave-pqr8 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for HTTP/127.3.186.212@KRBTEST.COM
I20260927 21:36:02.474138 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.455687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.3.186.254:40421)
0927 21:36:02.455846 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.455851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.455868 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:36:02.455924 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.455928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.455984 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.456069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:02.458365 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.459476 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.461676 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.463130 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.463272 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.466830 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:02.466837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:02.466842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:02.466878 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:02.470378 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.471188 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.471194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.471198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.471334 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.471937 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.471943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.471946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.472274 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:02.472512 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.473612 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.473829 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:36:02.479969 16690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:36:02.492362 16690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:36:02.585551 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.527090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.3.186.194:42147)
0927 21:36:02.558072 (+ 30982us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.558085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.575613 (+ 17528us) server_negotiation.cc:407] Connection header received
0927 21:36:02.576036 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.576048 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.576340 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.576524 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:02.577500 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.579104 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.582195 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.583743 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.583876 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.584103 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.584962 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.585151 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30776,"thread_start_us":31005,"threads_started":1}
I20260927 21:36:02.586397 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.526512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:42147 (local address 127.0.0.1:47928)
0927 21:36:02.575421 (+ 48909us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:02.575485 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:02.575734 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:02.576825 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:02.576833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:02.576866 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:02.577294 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.577308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.579286 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.579293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.581816 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.581830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.584355 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.584367 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.584679 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.584750 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 21:36:02.585828 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0927 21:36:02.586057 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48536,"thread_start_us":55414,"threads_started":1}
I20260927 21:36:02.590646 16966 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe3984a5ad141a9bf021230eb056e56 (DEFAULT_TABLE table=test-table [id=b0d9dd15c5b644b09eac85e30b8ced95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:02.592061 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe3984a5ad141a9bf021230eb056e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:02.600575 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.527754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:40389 (local address 127.0.0.1:35412)
0927 21:36:02.578158 (+ 50404us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:02.578175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:02.578296 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:02.580012 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:02.580022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:02.580065 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:02.580437 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.580449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.592059 (+ 11610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.592063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.593973 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.593987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.599798 (+ 5811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.599806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.599978 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.600026 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:36:02.600118 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:36:02.600254 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50290,"thread_start_us":66311,"threads_started":1}
I20260927 21:36:02.602784 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.539734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48316)
0927 21:36:02.581275 (+ 41541us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:02.581293 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:02.581408 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:02.584180 (+ 2772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:02.584184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:02.584210 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:02.584709 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.584720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.589423 (+ 4703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.589428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.596876 (+ 7448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.596888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.600913 (+ 4025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.600919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.601077 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.601106 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:36:02.601188 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:36:02.601328 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41393,"thread_start_us":37037,"threads_started":1}
I20260927 21:36:02.603569 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.540379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.3.186.193:41851)
0927 21:36:02.576430 (+ 36051us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.576440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.581540 (+ 5100us) server_negotiation.cc:407] Connection header received
0927 21:36:02.581637 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.581645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.581865 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.582044 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:02.584943 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.589230 (+ 4287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.597042 (+ 7812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.598442 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.598574 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.598783 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.603053 (+ 4270us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.603219 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35875,"thread_start_us":36226,"threads_started":1}
I20260927 21:36:02.604316 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.528462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.3.186.195:40389)
0927 21:36:02.574673 (+ 46211us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.574687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.578406 (+ 3719us) server_negotiation.cc:407] Connection header received
0927 21:36:02.578499 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.578506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.578710 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.578869 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:02.582404 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.591905 (+ 9501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.594289 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.599341 (+ 5052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.599463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.599661 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.603828 (+ 4167us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.603993 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45942,"thread_start_us":47981,"threads_started":1}
I20260927 21:36:02.608274 16821 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe3984a5ad141a9bf021230eb056e56 (DEFAULT_TABLE table=test-table [id=b0d9dd15c5b644b09eac85e30b8ced95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:02.609721 16821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe3984a5ad141a9bf021230eb056e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:02.609371 17100 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe3984a5ad141a9bf021230eb056e56 (DEFAULT_TABLE table=test-table [id=b0d9dd15c5b644b09eac85e30b8ced95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:02.610994 17100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe3984a5ad141a9bf021230eb056e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:02.613349 17188 tablet_bootstrap.cc:492] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Bootstrap starting.
I20260927 21:36:02.619133 17188 tablet_bootstrap.cc:654] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:02.620429 17188 log.cc:824] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:02.629336 17188 tablet_bootstrap.cc:492] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: No bootstrap required, opened a new log
I20260927 21:36:02.629617 17188 ts_tablet_manager.cc:1397] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260927 21:36:02.637612 17190 tablet_bootstrap.cc:492] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Bootstrap starting.
I20260927 21:36:02.641597 17188 raft_consensus.cc:348] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.642015 17188 raft_consensus.cc:374] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:02.642132 17188 raft_consensus.cc:729] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd662b5d73864a66ae5db2229442ee80, State: Initialized, Role: FOLLOWER
I20260927 21:36:02.642716 17190 tablet_bootstrap.cc:654] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:02.642820 17188 consensus_queue.cc:260] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.644023 17191 tablet_bootstrap.cc:492] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: Bootstrap starting.
I20260927 21:36:02.646442 17190 log.cc:824] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:02.647470 17191 tablet_bootstrap.cc:654] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:02.648674 17191 log.cc:824] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:02.649987 17031 heartbeater.cc:503] Master 127.3.186.254:40421 was elected leader, sending a full tablet report...
I20260927 21:36:02.651134 17188 ts_tablet_manager.cc:1428] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Time spent starting tablet: real 0.021s user 0.018s sys 0.001s
I20260927 21:36:02.656167 17190 tablet_bootstrap.cc:492] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: No bootstrap required, opened a new log
I20260927 21:36:02.656478 17190 ts_tablet_manager.cc:1397] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260927 21:36:02.664440 17190 raft_consensus.cc:348] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.664978 17190 raft_consensus.cc:374] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:02.665128 17190 raft_consensus.cc:729] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f46b6e714134d4db2979b4626d92f09, State: Initialized, Role: FOLLOWER
I20260927 21:36:02.665900 17190 consensus_queue.cc:260] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.673106 17191 tablet_bootstrap.cc:492] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: No bootstrap required, opened a new log
I20260927 21:36:02.673450 17191 ts_tablet_manager.cc:1397] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20260927 21:36:02.678220 17165 heartbeater.cc:503] Master 127.3.186.254:40421 was elected leader, sending a full tablet report...
I20260927 21:36:02.678916 17190 ts_tablet_manager.cc:1428] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Time spent starting tablet: real 0.022s user 0.004s sys 0.017s
I20260927 21:36:02.685576 17191 raft_consensus.cc:348] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.686017 17191 raft_consensus.cc:374] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:02.686125 17191 raft_consensus.cc:729] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9b2fd1117c43c9aedf9f3f92409e14, State: Initialized, Role: FOLLOWER
I20260927 21:36:02.686805 17191 consensus_queue.cc:260] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.687939 16887 heartbeater.cc:503] Master 127.3.186.254:40421 was elected leader, sending a full tablet report...
I20260927 21:36:02.688592 17191 ts_tablet_manager.cc:1428] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260927 21:36:02.695349 16888 tablet.cc:2369] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:36:02.754069 17032 tablet.cc:2369] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:36:02.761232 17195 raft_consensus.cc:482] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:36:02.761545 17195 raft_consensus.cc:504] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.763072 17195 leader_election.cc:288] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd662b5d73864a66ae5db2229442ee80 (127.3.186.194:42147), 9a9b2fd1117c43c9aedf9f3f92409e14 (127.3.186.193:41851)
I20260927 21:36:02.771140 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.763616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:48687 (local address 127.3.186.193:41851)
0927 21:36:02.763893 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.763902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.764364 (+ 462us) server_negotiation.cc:407] Connection header received
0927 21:36:02.764728 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.764735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.764815 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.764908 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:02.765757 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.766760 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.769060 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.770215 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.770352 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.770412 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.770787 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.770907 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 21:36:02.771253 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.763385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:42147 (local address 127.3.186.195:50851)
0927 21:36:02.764256 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:02.764274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:02.764409 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:02.764944 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:02.764948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:02.765002 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:02.765200 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.765214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.766666 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.766680 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.768784 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.768799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.770339 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.770347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.770584 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.770641 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:36:02.770748 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:36:02.770950 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":70}
I20260927 21:36:02.771492 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.763520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.3.186.195:48687)
0927 21:36:02.764263 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:02.764282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:02.764522 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:02.765122 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:02.765128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:02.765156 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:02.765579 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.765590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.766938 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.766945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.768913 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:02.768925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.770398 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.770403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:02.770628 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.770654 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:36:02.770748 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:36:02.770950 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":95,"thread_start_us":373,"threads_started":1}
I20260927 21:36:02.771596 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.763390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:50851 (local address 127.3.186.194:42147)
0927 21:36:02.763649 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.763659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.764532 (+ 873us) server_negotiation.cc:407] Connection header received
0927 21:36:02.764641 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.764648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.764796 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.764907 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:02.765393 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.766523 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.768952 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.770180 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.770338 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.770405 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.770787 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.770907 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:36:02.772428 16842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3984a5ad141a9bf021230eb056e56" candidate_uuid: "6f46b6e714134d4db2979b4626d92f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" is_pre_election: true
I20260927 21:36:02.772567 16986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3984a5ad141a9bf021230eb056e56" candidate_uuid: "6f46b6e714134d4db2979b4626d92f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd662b5d73864a66ae5db2229442ee80" is_pre_election: true
I20260927 21:36:02.772891 16842 raft_consensus.cc:2454] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f46b6e714134d4db2979b4626d92f09 in term 0.
I20260927 21:36:02.773053 16986 raft_consensus.cc:2454] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f46b6e714134d4db2979b4626d92f09 in term 0.
I20260927 21:36:02.773486 17055 leader_election.cc:302] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f46b6e714134d4db2979b4626d92f09, 9a9b2fd1117c43c9aedf9f3f92409e14; no voters:
I20260927 21:36:02.773855 17195 raft_consensus.cc:2781] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:36:02.773955 17195 raft_consensus.cc:482] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:36:02.774066 17195 raft_consensus.cc:3037] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:02.776067 17195 raft_consensus.cc:504] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.776595 17195 leader_election.cc:288] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [CANDIDATE]: Term 1 election: Requested vote from peers bd662b5d73864a66ae5db2229442ee80 (127.3.186.194:42147), 9a9b2fd1117c43c9aedf9f3f92409e14 (127.3.186.193:41851)
I20260927 21:36:02.777010 16986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3984a5ad141a9bf021230eb056e56" candidate_uuid: "6f46b6e714134d4db2979b4626d92f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd662b5d73864a66ae5db2229442ee80"
I20260927 21:36:02.777011 16842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3984a5ad141a9bf021230eb056e56" candidate_uuid: "6f46b6e714134d4db2979b4626d92f09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14"
I20260927 21:36:02.777174 16986 raft_consensus.cc:3037] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:02.777196 16842 raft_consensus.cc:3037] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:02.779183 16986 raft_consensus.cc:2454] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f46b6e714134d4db2979b4626d92f09 in term 1.
I20260927 21:36:02.779187 16842 raft_consensus.cc:2454] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f46b6e714134d4db2979b4626d92f09 in term 1.
I20260927 21:36:02.779551 17055 leader_election.cc:302] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f46b6e714134d4db2979b4626d92f09, 9a9b2fd1117c43c9aedf9f3f92409e14; no voters:
I20260927 21:36:02.779803 17195 raft_consensus.cc:2781] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:02.780161 17195 raft_consensus.cc:686] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 1 LEADER]: Becoming Leader. State: Replica: 6f46b6e714134d4db2979b4626d92f09, State: Running, Role: LEADER
I20260927 21:36:02.780645 17195 consensus_queue.cc:237] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } }
I20260927 21:36:02.784825 16689 catalog_manager.cc:5495] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f46b6e714134d4db2979b4626d92f09 (127.3.186.195). New cstate: current_term: 1 leader_uuid: "6f46b6e714134d4db2979b4626d92f09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f46b6e714134d4db2979b4626d92f09" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 } health_report { overall_health: UNKNOWN } } }
I20260927 21:36:02.828693 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.823815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.3.186.195:40389)
0927 21:36:02.823968 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.823972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.824023 (+ 51us) server_negotiation.cc:407] Connection header received
0927 21:36:02.824182 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.824186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.824272 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.824387 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:36:02.824770 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.825522 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.826873 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.827270 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.827355 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.827738 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:36:02.828159 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:36:02.828287 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.828383 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.828447 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:36:02.835567 16842 raft_consensus.cc:1261] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 1 FOLLOWER]: Refusing update from remote peer 6f46b6e714134d4db2979b4626d92f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:02.835764 16986 raft_consensus.cc:1261] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 1 FOLLOWER]: Refusing update from remote peer 6f46b6e714134d4db2979b4626d92f09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:02.836537 17205 consensus_queue.cc:1035] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd662b5d73864a66ae5db2229442ee80" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:02.836851 17195 consensus_queue.cc:1035] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9b2fd1117c43c9aedf9f3f92409e14" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 41851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:02.847020 17208 mvcc.cc:205] Tried to move back new op lower bound from 7334072167765774336 to 7334072167553351680. Current Snapshot: MvccSnapshot[applied={T|T < 7334072167765774336}]
I20260927 21:36:02.850526 17209 mvcc.cc:205] Tried to move back new op lower bound from 7334072167765774336 to 7334072167553351680. Current Snapshot: MvccSnapshot[applied={T|T < 7334072167765774336}]
I20260927 21:36:02.860690 16689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:02.860953 16689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:02.864171 16689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f932d6d97b684c11a882a1eb0fd00240: Sending DeleteTablet for 3 replicas of tablet 3fe3984a5ad141a9bf021230eb056e56
I20260927 21:36:02.865176 17100 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe3984a5ad141a9bf021230eb056e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:02 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:35412
I20260927 21:36:02.865326 16966 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe3984a5ad141a9bf021230eb056e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:02 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:47928
I20260927 21:36:02.865484 16821 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe3984a5ad141a9bf021230eb056e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:02 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:48316
I20260927 21:36:02.866082 17217 tablet_replica.cc:330] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: stopping tablet replica
I20260927 21:36:02.866616 17217 raft_consensus.cc:2229] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:36:02.867383 17217 raft_consensus.cc:2258] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:02.872130 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16727
I20260927 21:36:02.876734 17217 ts_tablet_manager.cc:1905] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:02.887488 17218 tablet_replica.cc:330] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: stopping tablet replica
I20260927 21:36:02.887852 17218 raft_consensus.cc:2229] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:02.890367 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:02.873167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.3.186.193:41851)
0927 21:36:02.873360 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:02.873368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:02.873390 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:36:02.873455 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:02.873464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:02.873581 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:02.873700 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:02.875504 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.876963 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.878760 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:02.879608 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:02.879761 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:02.882807 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:02.882817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:02.882832 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:02.882867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:02.887224 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.888166 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.888174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.888177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.888412 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:02.888843 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:02.888849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:02.888852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:02.889242 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:02.889606 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:02.889830 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 21:36:02.890031 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:36:02.890539 17218 raft_consensus.cc:2258] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:02.891006 16882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:02.892552 17218 ts_tablet_manager.cc:1905] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:02.897785 17217 ts_tablet_manager.cc:1918] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:02.897953 17217 log.cc:1196] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/3fe3984a5ad141a9bf021230eb056e56
I20260927 21:36:02.898469 17217 ts_tablet_manager.cc:1939] T 3fe3984a5ad141a9bf021230eb056e56 P 6f46b6e714134d4db2979b4626d92f09: Deleting consensus metadata
I20260927 21:36:02.900053 16675 catalog_manager.cc:4841] TS 6f46b6e714134d4db2979b4626d92f09 (127.3.186.195:40389): tablet 3fe3984a5ad141a9bf021230eb056e56 (table test-table [id=b0d9dd15c5b644b09eac85e30b8ced95]) successfully deleted
I20260927 21:36:02.901837 17218 ts_tablet_manager.cc:1918] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:02.901988 17218 log.cc:1196] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/3fe3984a5ad141a9bf021230eb056e56
I20260927 21:36:02.902460 17218 ts_tablet_manager.cc:1939] T 3fe3984a5ad141a9bf021230eb056e56 P bd662b5d73864a66ae5db2229442ee80: Deleting consensus metadata
I20260927 21:36:02.903968 16674 catalog_manager.cc:4841] TS bd662b5d73864a66ae5db2229442ee80 (127.3.186.194:42147): tablet 3fe3984a5ad141a9bf021230eb056e56 (table test-table [id=b0d9dd15c5b644b09eac85e30b8ced95]) successfully deleted
I20260927 21:36:03.050220 17219 tablet_replica.cc:330] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14: stopping tablet replica
I20260927 21:36:03.050616 17219 raft_consensus.cc:2229] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:03.051060 17219 raft_consensus.cc:2258] T 3fe3984a5ad141a9bf021230eb056e56 P 9a9b2fd1117c43c9aedf9f3f92409e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:03.052291 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16727
W20260927 21:36:03.069058 16676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:41851 (error 108)
I20260927 21:36:03.069202 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16889
I20260927 21:36:03.070976 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.070519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48346)
0927 21:36:03.070689 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.070804 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 21:36:03.071290 16676 catalog_manager.cc:4584] TS 9a9b2fd1117c43c9aedf9f3f92409e14 (127.3.186.193:41851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fe3984a5ad141a9bf021230eb056e56: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
I20260927 21:36:03.087664 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.070246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.3.186.194:42147)
0927 21:36:03.070463 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:03.070471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:03.070493 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:36:03.070574 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:03.070579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:03.070672 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:03.070773 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:03.072292 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:03.073133 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:03.074466 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:03.074867 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:03.075009 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:03.081919 (+ 6910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:03.081940 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:03.081953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:03.081985 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:03.084757 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:03.085455 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:03.085469 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:03.085471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:03.085682 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:03.086054 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:03.086060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:03.086064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:03.086422 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:03.086707 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:03.087005 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:36:03.087343 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:36:03.088375 17026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:03.124753 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.124290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48356)
0927 21:36:03.124484 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.124588 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 21:36:03.125938 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.125552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48358)
0927 21:36:03.125718 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.125797 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 21:36:03.175004 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16889
I20260927 21:36:03.186774 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.186171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48366)
0927 21:36:03.186422 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.186533 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:36:03.187988 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.187568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48378)
0927 21:36:03.187748 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.187839 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 21:36:03.191701 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17035
I20260927 21:36:03.206571 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.192419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.3.186.195:40389)
0927 21:36:03.192557 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:03.192561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:03.192752 (+ 191us) server_negotiation.cc:407] Connection header received
0927 21:36:03.192891 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:03.192898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:03.192980 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:03.193114 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:03.194262 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:03.195499 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:03.196738 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:03.197368 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:03.197653 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:03.199940 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:03.199952 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:03.199974 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:03.200024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:03.203739 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:03.204346 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:03.204355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:03.204359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:03.204553 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:03.204902 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:03.204912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:03.204915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:03.205348 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:03.205724 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:03.205984 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:36:03.206204 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:36:03.207288 17160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:03.268677 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.268135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48384)
0927 21:36:03.268411 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.268497 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 21:36:03.269745 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.269323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48388)
0927 21:36:03.269569 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.269644 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 21:36:03.323490 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17035
I20260927 21:36:03.342907 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 16657
I20260927 21:36:03.343947 16720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:03.451339 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 16657
I20260927 21:36:03.451370 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:03.450921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:41851 (local address 127.0.0.1:48400)
0927 21:36:03.451127 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:03.451224 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:41851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 21:36:03.472018 3819 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:36:03.806234 3819 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:36:03.806402 3819 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:36:04.150624 3819 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:36:04.150 UTC [15709] LOG: received smart shutdown request
2026-09-27 21:36:04.153 UTC [15709] LOG: background worker "logical replication launcher" (PID 15718) exited with exit code 1
2026-09-27 21:36:04.153 UTC [15713] LOG: shutting down
2026-09-27 21:36:04.312 UTC [15709] LOG: database system is shut down
2026-09-27T21:36:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48888 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:36:04 dist-test-slave-pqr8 krb5kdc[17233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:36:04 dist-test-slave-pqr8 krb5kdc[17233](info): set up 2 sockets
Sep 27 21:36:04 dist-test-slave-pqr8 krb5kdc[17233](info): commencing operation
krb5kdc: starting...
W20260927 21:36:05.505776 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:36:05 dist-test-slave-pqr8 krb5kdc[17233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:36:05.703236 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:35259
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:35259,127.3.186.253:33959,127.3.186.252:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:06.014982 17246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:06.015233 17246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:06.015316 17246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:06.022094 17246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:06.022184 17246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:06.022228 17246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:06.022284 17246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:06.022327 17246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:06.031289 17246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:35259,127.3.186.253:33959,127.3.186.252:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:06.032295 17246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:06.033404 17246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:06.044775 17251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:06.044822 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:06.044871 17252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:06.045704 17246 server_base.cc:1034] running on GCE node
I20260927 21:36:06.046368 17246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:06.047596 17246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:06.048822 17246 hybrid_clock.cc:648] HybridClock initialized: now 1790544966048786 us; error 60 us; skew 500 ppm
I20260927 21:36:06.049144 17246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:06.057106 17246 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:06.058529 17246 webserver.cc:460] Webserver started at http://127.3.186.254:33269/ using document root <none> and password file <none>
I20260927 21:36:06.059176 17246 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:06.059319 17246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:06.059617 17246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:06.062491 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "6166df354f434eb5a5e5703cf2076f79"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "9983b561adeee977d9b89512275b6f94"
server_key_iv: "e05e408d605fb4895ac5fa42de55f7f6"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.063277 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "6166df354f434eb5a5e5703cf2076f79"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "9983b561adeee977d9b89512275b6f94"
server_key_iv: "e05e408d605fb4895ac5fa42de55f7f6"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.068567 17246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:36:06.071763 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:06.073009 17246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:36:06.073140 17246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "6166df354f434eb5a5e5703cf2076f79"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "9983b561adeee977d9b89512275b6f94"
server_key_iv: "e05e408d605fb4895ac5fa42de55f7f6"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.073292 17246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:06.088711 17246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:06.092204 17246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:06.092404 17246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:06.109989 17246 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:35259
I20260927 21:36:06.110060 17312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:35259 every 8 connection(s)
I20260927 21:36:06.111209 17246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:06.114477 17313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:06.119957 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17246
I20260927 21:36:06.120198 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:36:06.120849 3819 external_mini_cluster.cc:1428] Setting key b3a99f4b87c4c35df392bf380d7145be
I20260927 21:36:06.120344 17313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.186.253@KRBTEST.COM; defaulting to no policy
I20260927 21:36:06.143553 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.123232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:35259 (local address 127.0.0.1:46742)
0927 21:36:06.124108 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.124161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:06.126964 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:06.127781 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:06.127790 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:06.128410 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:06.128743 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.128759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.130420 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.130429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.132372 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.132389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.133257 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.133266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.133478 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.134163 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:06.134196 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:06.136671 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:06.139793 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.139805 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.139814 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.140178 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.140568 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.140574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.140578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.140721 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.141363 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:06.141371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:06.141623 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 21:36:06.141860 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 21:36:06.142393 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":138,"thread_start_us":366,"threads_started":1}
I20260927 21:36:06.143599 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.123231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.3.186.254:35259)
0927 21:36:06.125040 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:06.125045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:06.125081 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:36:06.127332 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:06.127337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:06.127542 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:06.127689 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:06.128945 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.130267 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.132512 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.133026 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.133145 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.136897 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:06.136904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:06.136914 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:06.136939 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:06.139650 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.140302 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.140310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.140313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.140462 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.140821 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.140828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.140831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.141237 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:06.141464 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:06.141972 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 21:36:06.142265 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":1424,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.253@KRBTEST.COM" created.
I20260927 21:36:06.146464 17313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:36:06.147557 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.147228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50622)
0927 21:36:06.147370 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.147461 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 21:36:06.147821 17265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
I20260927 21:36:06.149318 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.148947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50636)
0927 21:36:06.149093 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.149151 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:36:06.149581 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:33959: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab.
Entry for principal kudu/127.3.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab.
I20260927 21:36:06.168316 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:33959
--webserver_interface=127.3.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab
--principal=kudu/127.3.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:35259,127.3.186.253:33959,127.3.186.252:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:06.198890 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } attempt: 1
I20260927 21:36:06.199939 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.199519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50650)
0927 21:36:06.199702 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.199818 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:36:06.200850 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.200562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50654)
0927 21:36:06.200684 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.200743 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 21:36:06.201196 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:33959: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
I20260927 21:36:06.269598 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } attempt: 2
I20260927 21:36:06.270717 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.270296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50658)
0927 21:36:06.270517 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.270598 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 21:36:06.271723 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.271425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50666)
0927 21:36:06.271543 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.271611 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:36:06.271950 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:33959: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
I20260927 21:36:06.363207 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } attempt: 3
I20260927 21:36:06.364445 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.363953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50674)
0927 21:36:06.364212 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.364323 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 21:36:06.365412 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.365103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50686)
0927 21:36:06.365238 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.365297 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:36:06.365657 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:33959: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
W20260927 21:36:06.471536 17320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:06.471848 17320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:06.472023 17320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:06.478976 17320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:06.479128 17320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:06.479224 17320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:06.479327 17320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:06.479414 17320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:06.488406 17320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:35259,127.3.186.253:33959,127.3.186.252:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab
--principal=kudu/127.3.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:33959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:06.489491 17320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:06.490662 17320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:06.501302 17325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:06.501562 17326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:06.501561 17328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:06.502472 17320 server_base.cc:1034] running on GCE node
I20260927 21:36:06.503207 17320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:06.504405 17320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:06.505607 17320 hybrid_clock.cc:648] HybridClock initialized: now 1790544966505561 us; error 61 us; skew 500 ppm
I20260927 21:36:06.505930 17320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:06.508724 17320 init.cc:377] Logged in from keytab as kudu/127.3.186.253@KRBTEST.COM (short username kudu)
I20260927 21:36:06.508903 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } attempt: 4
I20260927 21:36:06.509948 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.509434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50698)
0927 21:36:06.509751 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.509827 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 21:36:06.510246 17320 webserver.cc:460] Webserver started at http://127.3.186.253:39741/ using document root <none> and password file <none>
I20260927 21:36:06.510845 17320 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:06.510985 17320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:06.511006 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.510629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50708)
0927 21:36:06.510766 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.510862 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:36:06.511214 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:33959: Network error: Client connection negotiation failed: client connection to 127.3.186.253:33959: connect: Connection refused (error 111)
I20260927 21:36:06.511343 17320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:06.514041 17320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data/instance:
uuid: "a2e7b7b28fe74278b4f2ac21d007cb40"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "19e91d31adbc69393b6a74bfa31bb587"
server_key_iv: "eafd35100d3631b76ab13a666e6df2fe"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.514830 17320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal/instance:
uuid: "a2e7b7b28fe74278b4f2ac21d007cb40"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "19e91d31adbc69393b6a74bfa31bb587"
server_key_iv: "eafd35100d3631b76ab13a666e6df2fe"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.519404 17320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:36:06.522356 17335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:06.523656 17320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:36:06.523850 17320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal
uuid: "a2e7b7b28fe74278b4f2ac21d007cb40"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "19e91d31adbc69393b6a74bfa31bb587"
server_key_iv: "eafd35100d3631b76ab13a666e6df2fe"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.524037 17320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:06.550673 17320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:06.554159 17320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:06.554420 17320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:06.571933 17320 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.253:33959
I20260927 21:36:06.571967 17386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.253:33959 every 8 connection(s)
I20260927 21:36:06.573072 17320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
I20260927 21:36:06.576285 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17320
I20260927 21:36:06.576471 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-1/wal/instance
I20260927 21:36:06.576624 17387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:06.577050 3819 external_mini_cluster.cc:1428] Setting key 33c3371b8796431311405e9589319fad
I20260927 21:36:06.586002 17387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.186.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.252@KRBTEST.COM" created.
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:06.607175 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.588278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.3.186.254:35259)
0927 21:36:06.588433 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:06.588437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:06.589668 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 21:36:06.591417 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:06.591423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:06.591512 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:06.591625 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:06.592944 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.593632 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.595179 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.595599 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.595735 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.601330 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:06.601335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:06.601339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:06.601367 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:06.603909 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.604621 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.604628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.604644 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.604814 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.605237 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.605245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.605249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.605664 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:06.605838 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:06.606465 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 21:36:06.606705 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:36:06.607631 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.588802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:35259 (local address 127.0.0.1:46752)
0927 21:36:06.589467 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.589505 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:06.591157 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:06.591844 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:06.591852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:06.592306 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:06.592783 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.592797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.593800 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.593803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.595023 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.595034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.595812 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.595819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.596026 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.596678 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:06.596698 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:06.601096 (+ 4398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:06.604057 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.604065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.604077 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.604475 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.604944 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.604948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.604951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.605109 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.605800 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:06.605805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:06.606102 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 21:36:06.606337 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 21:36:06.606644 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":296,"threads_started":1}
I20260927 21:36:06.609223 17387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab.
Entry for principal kudu/127.3.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab.
I20260927 21:36:06.626962 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.609959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50724)
0927 21:36:06.610178 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.610202 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:06.610315 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:06.611625 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:06.611631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:06.612047 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:06.612165 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.612173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.614245 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.614249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.615057 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.615067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.616200 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.616204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.616336 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.616847 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:06.616862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:06.621195 (+ 4333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:06.624205 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.624208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.624211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.624650 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.625009 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.625012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.625015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.625246 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.625939 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:06.625948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:06.626056 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 21:36:06.626243 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 21:36:06.626490 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 21:36:06.626971 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.610205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.3.186.253:33959)
0927 21:36:06.610879 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:06.610887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:06.610918 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:36:06.610983 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:06.610991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:06.611225 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:06.611371 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:06.612337 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.614102 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.615204 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.616073 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.616213 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.621383 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:06.621393 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:06.621403 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:06.621430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:06.624072 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.624797 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.624802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.624804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.624908 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.625342 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.625346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.625348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.625815 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:06.626015 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:06.626359 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 21:36:06.626577 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":505,"thread_start_us":396,"threads_started":1}
I20260927 21:36:06.627004 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:38867
--webserver_interface=127.3.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab
--principal=kudu/127.3.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:35259,127.3.186.253:33959,127.3.186.252:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:06.629338 17387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:36:06.630360 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.629976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53268)
0927 21:36:06.630126 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.630238 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:36:06.630673 17339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.632053 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.631784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53280)
0927 21:36:06.631903 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.631970 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:36:06.632287 17387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.681571 17387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 1
I20260927 21:36:06.682572 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.682189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53296)
0927 21:36:06.682352 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.682442 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:36:06.683509 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.683234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53312)
0927 21:36:06.683363 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.683420 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:36:06.683759 17387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.751984 17387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 2
I20260927 21:36:06.753110 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.752697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53318)
0927 21:36:06.752883 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.752970 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:36:06.754094 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.753759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53330)
0927 21:36:06.753932 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.753994 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 21:36:06.754339 17387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.810427 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } attempt: 5
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:36:06.823669 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.811115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50726)
0927 21:36:06.811327 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.811348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:06.811485 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:06.811851 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:06.811858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:06.812230 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:06.812406 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.812415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.813366 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.813372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.814320 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:06.814332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.815060 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.815066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:06.815252 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.815873 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:06.815890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:06.817891 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:06.821552 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.821558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.821562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.821912 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.822279 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:06.822282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:06.822284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:06.822402 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:06.822965 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:06.822968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:06.823078 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 21:36:06.823180 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:36:06.823431 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 21:36:06.823853 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.811212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.3.186.253:33959)
0927 21:36:06.811392 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:06.811400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:06.811420 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:36:06.811644 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:06.811648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:06.811708 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:06.811813 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:06.812550 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.813238 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.814460 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:06.814911 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:06.815047 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:06.818099 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:06.818107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:06.818115 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:06.818161 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:06.821370 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.822022 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.822028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.822031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.822174 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:06.822511 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:06.822516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:06.822519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:06.822860 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:06.823001 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:06.823349 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 21:36:06.823555 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:36:06.824990 17313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:36:06.825963 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.825646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53340)
0927 21:36:06.825780 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.825848 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:36:06.826788 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.826490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53346)
0927 21:36:06.826632 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.826707 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 21:36:06.826996 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.845573 17387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 3
I20260927 21:36:06.846468 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.846099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53356)
0927 21:36:06.846269 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.846342 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 21:36:06.847450 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.847004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53366)
0927 21:36:06.847257 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.847323 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260927 21:36:06.847659 17387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.878198 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 1
I20260927 21:36:06.879269 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.878745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53382)
0927 21:36:06.878960 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.879079 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 21:36:06.880265 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.879892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53386)
0927 21:36:06.880042 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.880134 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:36:06.880491 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
W20260927 21:36:06.934742 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:06.935010 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:06.935174 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:06.941851 17394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:06.941968 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:06.942044 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:06.942140 17394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:06.942219 17394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:06.948724 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 2
I20260927 21:36:06.949796 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.949269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53400)
0927 21:36:06.949496 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.949623 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 21:36:06.950762 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.950402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53404)
0927 21:36:06.950549 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.950633 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:36:06.950977 17313 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:06.952044 17394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:35259,127.3.186.253:33959,127.3.186.252:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab
--principal=kudu/127.3.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:06.953055 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:06.954276 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:06.965215 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:06.965171 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:06.965262 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:06.971052 17394 server_base.cc:1034] running on GCE node
I20260927 21:36:06.971668 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:06.972754 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:06.973975 17394 hybrid_clock.cc:648] HybridClock initialized: now 1790544966973933 us; error 51 us; skew 500 ppm
I20260927 21:36:06.974318 17394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:06 dist-test-slave-pqr8 krb5kdc[17233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:06.977264 17394 init.cc:377] Logged in from keytab as kudu/127.3.186.252@KRBTEST.COM (short username kudu)
I20260927 21:36:06.978551 17394 webserver.cc:460] Webserver started at http://127.3.186.252:38937/ using document root <none> and password file <none>
I20260927 21:36:06.979153 17394 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:06.979247 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:06.979465 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:06.982153 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data/instance:
uuid: "c28d430211d34ec39fc7f6cbf6996119"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "f3952975984bd3e9235f41e48cd14789"
server_key_iv: "85baf4ede639e7da1a723f9623958b84"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.982964 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal/instance:
uuid: "c28d430211d34ec39fc7f6cbf6996119"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "f3952975984bd3e9235f41e48cd14789"
server_key_iv: "85baf4ede639e7da1a723f9623958b84"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.987691 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 21:36:06.990787 17409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:06.990864 17387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 4
I20260927 21:36:06.991773 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.991375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53414)
0927 21:36:06.991568 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.991639 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 21:36:06.992064 17394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:36:06.992206 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal
uuid: "c28d430211d34ec39fc7f6cbf6996119"
format_stamp: "Formatted at 2026-09-27 21:36:06 on dist-test-slave-pqr8"
server_key: "f3952975984bd3e9235f41e48cd14789"
server_key_iv: "85baf4ede639e7da1a723f9623958b84"
server_key_version: "encryptionkey@0"
I20260927 21:36:06.992375 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:06.992615 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:06.992322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53430)
0927 21:36:06.992448 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:06.992511 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 21:36:06.992796 17387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:38867: Network error: Client connection negotiation failed: client connection to 127.3.186.252:38867: connect: Connection refused (error 111)
I20260927 21:36:07.011204 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:07.014645 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:07.014902 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:07.031591 17394 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.252:38867
I20260927 21:36:07.031667 17460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.252:38867 every 8 connection(s)
I20260927 21:36:07.032728 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
I20260927 21:36:07.036093 17461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:07.036226 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17394
I20260927 21:36:07.036420 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544773327888-3819-0/minicluster-data/master-2/wal/instance
I20260927 21:36:07.036976 3819 external_mini_cluster.cc:1428] Setting key d9bf035fb261f9c309756bcea6fb6da3
I20260927 21:36:07.044749 17461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:07.057183 17313 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 3
I20260927 21:36:07.059871 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.038793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.3.186.252:38867)
0927 21:36:07.039629 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.039639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.039675 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:36:07.041813 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.041832 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.042166 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.042465 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.044217 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.045450 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.047334 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.048396 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.048561 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.052638 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.052648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.052657 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.052690 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.055727 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.056380 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.056387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.056392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.056524 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.056897 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.056903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.056906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.057353 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.057735 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.058171 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.058566 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":457,"threads_started":1}
I20260927 21:36:07.062443 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.047266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.3.186.254:35259)
0927 21:36:07.047476 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.047482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.048179 (+ 697us) server_negotiation.cc:407] Connection header received
0927 21:36:07.048345 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.048349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.048407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.048515 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.049742 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.050402 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.051968 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.052385 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.052517 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.055958 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.055963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.055966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.055996 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.059215 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.060143 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.060151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.060158 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.060319 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.060760 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.060765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.060769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.061070 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.061221 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.061837 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.062038 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:36:07.062444 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.047295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:35259 (local address 127.0.0.1:46762)
0927 21:36:07.048036 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:07.048073 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:07.048219 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:07.048569 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:07.048576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:07.049275 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:07.049596 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.049606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.050550 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.050555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.051837 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.051848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.052625 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.052633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.052779 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.053393 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:07.053417 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:07.055777 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:07.059328 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.059339 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.059347 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.059875 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.060452 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.060457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.060460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.060622 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.061286 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:07.061293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:07.061491 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 21:36:07.061738 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 21:36:07.062031 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":1,"thread_start_us":338,"threads_started":1}
I20260927 21:36:07.064307 17461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:36:07.074194 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.058052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53450)
0927 21:36:07.058405 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:07.058425 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:07.058530 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:07.059414 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:07.059418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:07.059811 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:07.060043 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.060054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.061602 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.061607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.063043 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.063054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.063894 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.063899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.064057 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.064644 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:07.064688 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:07.067448 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:07.071535 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.071540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.071544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.072040 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.072409 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.072412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.072414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.072586 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.073195 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:07.073201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:07.073374 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 21:36:07.073568 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 21:36:07.073800 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260927 21:36:07.074239 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.058442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.3.186.252:38867)
0927 21:36:07.059144 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.059149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.059173 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:36:07.059219 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.059223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.059281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.059383 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.060243 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.061448 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.063179 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.063751 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.063879 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.067658 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.067664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.067670 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.067703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.071384 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.072159 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.072165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.072170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.072305 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.072696 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.072701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.072705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.073020 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.073220 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.073691 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.073873 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":395,"threads_started":1}
I20260927 21:36:07.080018 17313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: Bootstrap starting.
I20260927 21:36:07.080384 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.065694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.3.186.253:33959)
0927 21:36:07.065980 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.065984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.065999 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:36:07.066043 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.066046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.066098 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.066250 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.066998 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.067755 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.069311 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.069906 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.070038 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.073946 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.073952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.073958 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.073997 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.077777 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.078432 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.078439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.078442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.078610 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.079079 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.079085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.079089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.079381 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.079527 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.079915 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.080053 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260927 21:36:07.080408 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.064926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:33959 (local address 127.0.0.1:50736)
0927 21:36:07.065077 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:07.065094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:07.065187 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:07.066285 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:07.066289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:07.066683 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:07.066847 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.066858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.067916 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.067923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.069169 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.069185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.070043 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.070048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.070286 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.070797 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:07.070813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:07.073754 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:07.077926 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.077931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.077934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.078308 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.078754 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.078758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.078762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.078927 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.079524 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:07.079528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:07.079692 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 21:36:07.079836 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:36:07.080067 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:36:07.081301 17461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:36:07.082147 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.063247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.3.186.254:35259)
0927 21:36:07.063385 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.063389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.063400 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:36:07.063432 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.063435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.063519 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.063619 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.065176 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.066333 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.067979 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.068599 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.068715 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.074098 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.074116 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.074120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.074152 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.077777 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.078509 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.078514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.078517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.078662 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.079023 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.079029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.079032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.079381 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.079662 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.081489 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.081783 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:36:07.084424 17313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:07.085817 17313 log.cc:824] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:36:07.089002 17313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: No bootstrap required, opened a new log
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:36:07.095441 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.081938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53466)
0927 21:36:07.082085 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:07.082108 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:07.082192 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:07.082626 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:07.082631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:07.083050 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:07.083223 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.083234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.084610 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.084615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.085419 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.085429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.086466 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.086471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.086673 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.087257 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:07.087278 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:07.089663 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:07.093124 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.093129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.093134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.093458 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.093858 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.093863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.093867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.093988 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.094429 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:07.094434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:07.094604 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 21:36:07.094760 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 21:36:07.095064 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 21:36:07.095657 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.082145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.3.186.252:38867)
0927 21:36:07.082264 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.082268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.082283 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:36:07.082410 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.082414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.082485 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.082576 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.083361 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.084443 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.085579 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.086219 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.086332 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.089886 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.089891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.089896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.089931 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.093011 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.093628 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.093633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.093639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.093757 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.094091 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.094095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.094098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.094322 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.094471 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.094881 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.095291 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":43}
I20260927 21:36:07.099179 17313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.099558 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.083702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.3.186.253:33959)
0927 21:36:07.083854 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.083859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.083876 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:36:07.083988 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.083993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.084050 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.084139 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.085702 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.086781 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.088172 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.088583 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.088721 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.093955 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.093961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.093966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.093999 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.097132 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.097819 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.097824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.097827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.097955 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.098306 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.098311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.098314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.098548 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.098804 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.099068 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.099286 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:36:07.099642 17313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:07.099797 17313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6166df354f434eb5a5e5703cf2076f79, State: Initialized, Role: FOLLOWER
I20260927 21:36:07.100881 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119: Bootstrap starting.
I20260927 21:36:07.100818 17313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.101928 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.102404 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.103058 17313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:07.103679 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:36:07.104427 17461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:07.105887 17461 log.cc:824] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:07.114466 17481 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:36:07.118866 17246 master_runner.cc:186] Error getting master registration for 127.3.186.253:33959: OK, instance_id { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" instance_seqno: 1790544966566929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:36:07.131918 17246 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:36:07.135864 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119: No bootstrap required, opened a new log
I20260927 21:36:07.149077 17461 raft_consensus.cc:348] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.149585 17461 raft_consensus.cc:374] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:07.149762 17461 raft_consensus.cc:729] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28d430211d34ec39fc7f6cbf6996119, State: Initialized, Role: FOLLOWER
I20260927 21:36:07.150555 17461 consensus_queue.cc:260] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.154143 17488 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.154613 17488 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.155525 17461 sys_catalog.cc:564] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:07.157289 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.116230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.3.186.254:35259)
0927 21:36:07.116368 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.116373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.123804 (+ 7431us) server_negotiation.cc:407] Connection header received
0927 21:36:07.124930 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.124935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.125014 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.125085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.141762 (+ 16677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.142827 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.144958 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.145718 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.145814 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.149612 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.149619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.149626 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.149674 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.153126 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.154908 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.154916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.154922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.155095 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.155545 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.155551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.155554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.155906 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.156140 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.156725 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.156935 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:36:07.159668 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.115931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.3.186.253:33959)
0927 21:36:07.116076 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.116082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.141960 (+ 25878us) server_negotiation.cc:407] Connection header received
0927 21:36:07.142102 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.142108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.142187 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.142282 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.144335 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.145407 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.147665 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.148303 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.148388 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.151802 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.151811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.151819 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.151868 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.155590 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.156492 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.156499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.156503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.156652 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.157159 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.157167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.157173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.157546 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.157844 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.159090 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.159322 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:36:07.161073 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.124201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.3.186.252:38867)
0927 21:36:07.124339 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.124345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.142494 (+ 18149us) server_negotiation.cc:407] Connection header received
0927 21:36:07.144042 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.144047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.144109 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.144171 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.146442 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.147673 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.149950 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.150701 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.150805 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.154180 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.154185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.154188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.154224 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.157758 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.158603 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.158608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.158611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.158764 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.159215 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.159220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.159222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.159596 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.159837 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.160378 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.160619 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 21:36:07.167583 17499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:36:07.167718 17499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:36:07.169215 17394 master_runner.cc:186] Error getting master registration for 127.3.186.253:33959: OK, instance_id { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" instance_seqno: 1790544966566929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:36:07.177187 17394 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:36:07.181173 17488 raft_consensus.cc:482] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:36:07.181437 17488 raft_consensus.cc:504] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.182838 17488 leader_election.cc:288] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6166df354f434eb5a5e5703cf2076f79 (127.3.186.254:35259), a2e7b7b28fe74278b4f2ac21d007cb40 (127.3.186.253:33959)
I20260927 21:36:07.183507 17288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28d430211d34ec39fc7f6cbf6996119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6166df354f434eb5a5e5703cf2076f79" is_pre_election: true
I20260927 21:36:07.184013 17288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28d430211d34ec39fc7f6cbf6996119 in term 0.
I20260927 21:36:07.184572 17412 leader_election.cc:302] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6166df354f434eb5a5e5703cf2076f79, c28d430211d34ec39fc7f6cbf6996119; no voters:
I20260927 21:36:07.184967 17488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:36:07.185111 17488 raft_consensus.cc:482] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:36:07.185221 17488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:07.188434 17488 raft_consensus.cc:504] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.189086 17488 leader_election.cc:288] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [CANDIDATE]: Term 1 election: Requested vote from peers 6166df354f434eb5a5e5703cf2076f79 (127.3.186.254:35259), a2e7b7b28fe74278b4f2ac21d007cb40 (127.3.186.253:33959)
I20260927 21:36:07.189810 17288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28d430211d34ec39fc7f6cbf6996119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6166df354f434eb5a5e5703cf2076f79"
I20260927 21:36:07.189998 17288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:07.192926 17288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28d430211d34ec39fc7f6cbf6996119 in term 1.
I20260927 21:36:07.193316 17412 leader_election.cc:302] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6166df354f434eb5a5e5703cf2076f79, c28d430211d34ec39fc7f6cbf6996119; no voters:
I20260927 21:36:07.193667 17488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:07.194460 17488 raft_consensus.cc:686] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [term 1 LEADER]: Becoming Leader. State: Replica: c28d430211d34ec39fc7f6cbf6996119, State: Running, Role: LEADER
I20260927 21:36:07.195032 17488 consensus_queue.cc:237] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.198158 17501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28d430211d34ec39fc7f6cbf6996119. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.198546 17501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:07.199429 17503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:07.201300 17503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:07.209810 17288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [term 1 FOLLOWER]: Refusing update from remote peer c28d430211d34ec39fc7f6cbf6996119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:07.210687 17501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:07.220181 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28d430211d34ec39fc7f6cbf6996119. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.220489 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.224787 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.224998 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.226418 17501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.226653 17501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:07.228554 17503 catalog_manager.cc:1347] Generated new cluster ID: d8affccfe6fe40b0bf5aa334eca0db81
I20260927 21:36:07.228654 17503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:07.245031 17503 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:36:07.247658 17503 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:07.261067 17503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119: Generated new TSK 0
I20260927 21:36:07.262177 17503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:36:07.291991 17387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } attempt: 5
Sep 27 21:36:07 dist-test-slave-pqr8 krb5kdc[17233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544966, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:36:07.309471 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.292639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.3.186.252:38867)
0927 21:36:07.292783 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:07.292789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:07.292806 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:36:07.292952 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:07.292957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:07.293022 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:07.293129 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:07.294090 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.295290 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.297208 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.297765 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.297901 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.301690 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:07.301699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:07.301703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:07.301743 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:07.306716 (+ 4973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.307455 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.307460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.307464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.307607 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:07.308009 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:07.308014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:07.308017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:07.308373 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:07.308510 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:07.308918 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 21:36:07.309115 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:36:07.309484 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:07.292481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:38867 (local address 127.0.0.1:53480)
0927 21:36:07.292707 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:07.292725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:07.292823 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:07.293155 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:07.293160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:07.293693 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:07.293917 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.293930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.295418 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.295423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.297061 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:07.297075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:07.297906 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:07.297911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:07.298112 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:07.298702 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:07.298734 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:07.301484 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:07.306826 (+ 5342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.306831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.306835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.307337 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.307739 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:07.307744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:07.307748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:07.307904 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:07.308493 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:07.308498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:07.308657 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 21:36:07.308812 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 21:36:07.309094 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 21:36:07.315685 17387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: Bootstrap starting.
I20260927 21:36:07.319597 17387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:07.320878 17387 log.cc:824] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:07.330080 17387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: No bootstrap required, opened a new log
I20260927 21:36:07.341014 17387 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.341461 17387 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:07.341609 17387 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e7b7b28fe74278b4f2ac21d007cb40, State: Initialized, Role: FOLLOWER
I20260927 21:36:07.342238 17387 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } }
I20260927 21:36:07.343196 17513 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.343532 17513 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.344308 17387 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:07.344358 17362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28d430211d34ec39fc7f6cbf6996119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" is_pre_election: true
I20260927 21:36:07.344960 17362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28d430211d34ec39fc7f6cbf6996119 in term 0.
I20260927 21:36:07.345350 17361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c28d430211d34ec39fc7f6cbf6996119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40"
I20260927 21:36:07.345610 17361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:07.349045 17361 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28d430211d34ec39fc7f6cbf6996119 in term 1.
I20260927 21:36:07.349835 17360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [term 1 FOLLOWER]: Refusing update from remote peer c28d430211d34ec39fc7f6cbf6996119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:07.350677 17488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:07.376677 17513 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c28d430211d34ec39fc7f6cbf6996119. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.376644 17523 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.376986 17513 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.377027 17523 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:36:07.384465 17530 catalog_manager.cc:1259] Loaded cluster ID: d8affccfe6fe40b0bf5aa334eca0db81
I20260927 21:36:07.384593 17530 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: loading cluster ID for follower catalog manager: success
I20260927 21:36:07.388438 17530 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: acquiring CA information for follower catalog manager: success
I20260927 21:36:07.390746 17530 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2e7b7b28fe74278b4f2ac21d007cb40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:36:07.902173 17488 sys_catalog.cc:455] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c28d430211d34ec39fc7f6cbf6996119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6166df354f434eb5a5e5703cf2076f79" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35259 } } peers { permanent_uuid: "a2e7b7b28fe74278b4f2ac21d007cb40" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 33959 } } peers { permanent_uuid: "c28d430211d34ec39fc7f6cbf6996119" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 38867 } } }
I20260927 21:36:07.902510 17488 sys_catalog.cc:458] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:08.119335 17481 catalog_manager.cc:1259] Loaded cluster ID: d8affccfe6fe40b0bf5aa334eca0db81
I20260927 21:36:08.119441 17481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: loading cluster ID for follower catalog manager: success
I20260927 21:36:08.123052 17481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: acquiring CA information for follower catalog manager: success
I20260927 21:36:08.125432 17481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6166df354f434eb5a5e5703cf2076f79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:36:08.345860 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17246
I20260927 21:36:08.347450 17308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:08.458019 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17246
I20260927 21:36:08.477326 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17320
I20260927 21:36:08.478497 17382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:08.579578 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17320
I20260927 21:36:08.584390 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:08.583636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:35259 (local address 127.0.0.1:46796)
0927 21:36:08.584111 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:08.584233 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":266,"threads_started":1}
W20260927 21:36:08.584702 17412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.254:35259: connect: Connection refused (error 111)
I20260927 21:36:08.586222 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:08.585841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:35259 (local address 127.0.0.1:46804)
0927 21:36:08.586020 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:08.586098 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:35259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 21:36:08.586421 17412 consensus_peers.cc:487] T 00000000000000000000000000000000 P c28d430211d34ec39fc7f6cbf6996119 -> Peer 6166df354f434eb5a5e5703cf2076f79 (127.3.186.254:35259): Couldn't send request to peer 6166df354f434eb5a5e5703cf2076f79. Status: Network error: Client connection negotiation failed: client connection to 127.3.186.254:35259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:36:08.596262 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17394
I20260927 21:36:08.597214 17456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:08.699719 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17394
2026-09-27T21:36:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4261 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:36:08 dist-test-slave-pqr8 krb5kdc[17543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:36:08 dist-test-slave-pqr8 krb5kdc[17543](info): set up 2 sockets
Sep 27 21:36:08 dist-test-slave-pqr8 krb5kdc[17543](info): commencing operation
krb5kdc: starting...
W20260927 21:36:09.765949 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:36:09 dist-test-slave-pqr8 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:36:09.972689 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:35297
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:10.275869 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:10.276117 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:10.276275 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:10.282941 17556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:10.283028 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:10.283149 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:10.283262 17556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:10.283322 17556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:10.292515 17556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:35297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:10.293704 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:10.294911 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:10.306236 17564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:10.306325 17561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:10.306281 17562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:10.307461 17556 server_base.cc:1034] running on GCE node
I20260927 21:36:10.308166 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:10.309355 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:10.310580 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790544970310530 us; error 35 us; skew 500 ppm
I20260927 21:36:10.310899 17556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:10 dist-test-slave-pqr8 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544970, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:10.313807 17556 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:10.315194 17556 webserver.cc:460] Webserver started at http://127.3.186.254:45321/ using document root <none> and password file <none>
I20260927 21:36:10.315765 17556 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:10.315860 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:10.316094 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:10.318822 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "3afde07f0f174da29d1fab74a7c4d9f6"
format_stamp: "Formatted at 2026-09-27 21:36:10 on dist-test-slave-pqr8"
server_key: "a949fb1fdc91c0141120801534b802d0"
server_key_iv: "3dda6507cc0db828d9474b595ea3b81d"
server_key_version: "encryptionkey@0"
I20260927 21:36:10.319470 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "3afde07f0f174da29d1fab74a7c4d9f6"
format_stamp: "Formatted at 2026-09-27 21:36:10 on dist-test-slave-pqr8"
server_key: "a949fb1fdc91c0141120801534b802d0"
server_key_iv: "3dda6507cc0db828d9474b595ea3b81d"
server_key_version: "encryptionkey@0"
I20260927 21:36:10.324055 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 21:36:10.327180 17571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:10.328383 17556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:36:10.328503 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "3afde07f0f174da29d1fab74a7c4d9f6"
format_stamp: "Formatted at 2026-09-27 21:36:10 on dist-test-slave-pqr8"
server_key: "a949fb1fdc91c0141120801534b802d0"
server_key_iv: "3dda6507cc0db828d9474b595ea3b81d"
server_key_version: "encryptionkey@0"
I20260927 21:36:10.328645 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:10.347203 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:10.351454 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:10.351660 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:10.369179 17556 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:35297
I20260927 21:36:10.369206 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:35297 every 8 connection(s)
I20260927 21:36:10.370437 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:10.373888 17623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:10.378798 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17556
I20260927 21:36:10.379019 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:36:10.379748 3819 external_mini_cluster.cc:1428] Setting key 8363d135f6bbea3e3b0aaa3f1e9228fa
I20260927 21:36:10.384640 17623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6: Bootstrap starting.
I20260927 21:36:10.388151 17623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:10.389297 17623 log.cc:824] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:10.392426 17623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6: No bootstrap required, opened a new log
Sep 27 21:36:10 dist-test-slave-pqr8 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:10.402887 17623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35297 } }
I20260927 21:36:10.403311 17623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:10.403412 17623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afde07f0f174da29d1fab74a7c4d9f6, State: Initialized, Role: FOLLOWER
I20260927 21:36:10.403780 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:10.381630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.3.186.254:35297)
0927 21:36:10.382419 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:10.382429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:10.382470 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:36:10.384115 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:10.384136 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:10.384471 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:10.384846 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:10.386590 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:10.387863 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:10.389812 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:10.390833 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:10.390949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:10.395568 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:10.395578 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:10.395596 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:10.395637 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:10.399568 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:10.400232 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:10.400244 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:10.400251 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:10.400452 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:10.400806 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:10.400812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:10.400817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:10.401480 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:10.401801 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:10.402105 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:36:10.402482 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":361,"threads_started":1}
I20260927 21:36:10.404036 17623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35297 } }
I20260927 21:36:10.404301 17623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:10.404487 17623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:10.404678 17623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:10.407106 17623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35297 } }
I20260927 21:36:10.407533 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:36:10.407629 17623 leader_election.cc:302] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3afde07f0f174da29d1fab74a7c4d9f6; no voters:
I20260927 21:36:10.408174 17623 leader_election.cc:288] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:36:10.408284 17628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:10.408857 17628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [term 1 LEADER]: Becoming Leader. State: Replica: 3afde07f0f174da29d1fab74a7c4d9f6, State: Running, Role: LEADER
I20260927 21:36:10.409348 17628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35297 } }
I20260927 21:36:10.411787 17630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35297 } } }
I20260927 21:36:10.412147 17630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:10.413411 17633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3afde07f0f174da29d1fab74a7c4d9f6. Latest consensus state: current_term: 1 leader_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afde07f0f174da29d1fab74a7c4d9f6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35297 } } }
I20260927 21:36:10.413817 17633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:10.416054 17623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:10.416217 17632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:10.421883 17632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:10.431195 17632 catalog_manager.cc:1347] Generated new cluster ID: 545b2a7ff4b040b69e5b7c1e0eafcab0
I20260927 21:36:10.431300 17632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:10.454295 17632 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:36:10.455470 17632 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:10.464320 17632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3afde07f0f174da29d1fab74a7c4d9f6: Generated new TSK 0
I20260927 21:36:10.465205 17632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:36:10.520318 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17556
I20260927 21:36:10.521373 17618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:10.614362 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17556
2026-09-27T21:36:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1910 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:36:10 dist-test-slave-pqr8 krb5kdc[17652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:36:10 dist-test-slave-pqr8 krb5kdc[17652](info): set up 2 sockets
Sep 27 21:36:10 dist-test-slave-pqr8 krb5kdc[17652](info): commencing operation
krb5kdc: starting...
W20260927 21:36:11.662014 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:36:11 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:36:11.862037 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44503
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:12.163283 17665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:12.163563 17665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:12.163712 17665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:12.170326 17665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:12.170449 17665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:12.170549 17665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:12.170641 17665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:12.170699 17665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:12.180016 17665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:12.181110 17665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:12.182297 17665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:12.193270 17670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:12.193318 17673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:12.193360 17671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:12.194217 17665 server_base.cc:1034] running on GCE node
I20260927 21:36:12.194880 17665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:12.196120 17665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:12.197314 17665 hybrid_clock.cc:648] HybridClock initialized: now 1790544972197263 us; error 47 us; skew 500 ppm
I20260927 21:36:12.197706 17665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:12 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:12.208254 17665 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:12.209677 17665 webserver.cc:460] Webserver started at http://127.3.186.254:40311/ using document root <none> and password file <none>
I20260927 21:36:12.210265 17665 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:12.210389 17665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:12.210676 17665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:12.213225 17665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "2b409676f4b74d888c62f4516747ece3"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "fcd24e7b09abe48e0a98e0d3b82dd2be"
server_key_iv: "0fcd104b7a2e37aa75c68b470a99414b"
server_key_version: "encryptionkey@0"
I20260927 21:36:12.213909 17665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "2b409676f4b74d888c62f4516747ece3"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "fcd24e7b09abe48e0a98e0d3b82dd2be"
server_key_iv: "0fcd104b7a2e37aa75c68b470a99414b"
server_key_version: "encryptionkey@0"
I20260927 21:36:12.218480 17665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:36:12.221410 17680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:12.222613 17665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:36:12.222787 17665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "2b409676f4b74d888c62f4516747ece3"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "fcd24e7b09abe48e0a98e0d3b82dd2be"
server_key_iv: "0fcd104b7a2e37aa75c68b470a99414b"
server_key_version: "encryptionkey@0"
I20260927 21:36:12.222962 17665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:12.244216 17665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:12.247627 17665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:12.247896 17665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:12.265231 17665 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44503
I20260927 21:36:12.265303 17731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44503 every 8 connection(s)
I20260927 21:36:12.266403 17665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:12.267625 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17665
I20260927 21:36:12.267856 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:36:12.268677 3819 external_mini_cluster.cc:1428] Setting key d6f864512381cea420b2caf99207f894
I20260927 21:36:12.270712 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:36:12 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:12.283490 17732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap starting.
I20260927 21:36:12.286623 17732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:12.287623 17732 log.cc:824] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:12.290184 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:12.270499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.3.186.254:44503)
0927 21:36:12.271195 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:12.271205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:12.271238 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:36:12.272650 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:12.272667 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:12.272985 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:12.273278 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:12.274883 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:12.276126 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:12.278023 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:12.278598 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:12.278719 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:12.283533 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:12.283541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:12.283552 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:12.283583 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:12.286409 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:12.287163 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:12.287169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:12.287174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:12.287318 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:12.287725 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:12.287729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:12.287731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:12.288255 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:12.288590 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:12.288939 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 21:36:12.289301 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":349,"threads_started":1}
I20260927 21:36:12.290223 17732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: No bootstrap required, opened a new log
I20260927 21:36:12.297595 17732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:12.297922 17732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:12.298030 17732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b409676f4b74d888c62f4516747ece3, State: Initialized, Role: FOLLOWER
I20260927 21:36:12.298532 17732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:12.298777 17732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:12.298920 17732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:12.299088 17732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:12.301170 17732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:12.301683 17732 leader_election.cc:302] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b409676f4b74d888c62f4516747ece3; no voters:
I20260927 21:36:12.302222 17732 leader_election.cc:288] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:36:12.302314 17737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:12.303012 17737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 LEADER]: Becoming Leader. State: Replica: 2b409676f4b74d888c62f4516747ece3, State: Running, Role: LEADER
I20260927 21:36:12.303493 17737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:12.304092 17732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:12.306061 17739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b409676f4b74d888c62f4516747ece3. Latest consensus state: current_term: 1 leader_uuid: "2b409676f4b74d888c62f4516747ece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } } }
I20260927 21:36:12.306083 17738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b409676f4b74d888c62f4516747ece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } } }
I20260927 21:36:12.306447 17739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:12.306447 17738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:12.308444 17743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:12.313202 17743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:12.321849 17743 catalog_manager.cc:1347] Generated new cluster ID: f90f3c0816e5425eb7eb41c08c680de4
I20260927 21:36:12.321942 17743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:12.335258 17743 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:36:12.336424 17743 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:12.344763 17743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Generated new TSK 0
I20260927 21:36:12.345407 17743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:36:12.401355 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44503
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:12.704623 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:12.704914 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:12.705118 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:12.711874 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:12.712030 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:12.712181 17758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:36:12.720839 17758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:12.721949 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:12.723150 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:12.735543 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:12.735663 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:12.735615 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:12.736294 17758 server_base.cc:1034] running on GCE node
I20260927 21:36:12.736908 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:12.737840 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:12.739025 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790544972738993 us; error 30 us; skew 500 ppm
I20260927 21:36:12.739320 17758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:12 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:12.742305 17758 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:36:12.743602 17758 webserver.cc:460] Webserver started at http://127.3.186.193:43987/ using document root <none> and password file <none>
I20260927 21:36:12.744200 17758 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:12.744325 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:12.744644 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:12.747347 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "94848b97740e407fb3bf19da014b1783"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "06a4a59bc6f2b0a84131ece231fea92b"
server_key_iv: "6d76a22b831125b03c11ee92c33899e6"
server_key_version: "encryptionkey@0"
I20260927 21:36:12.748050 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "94848b97740e407fb3bf19da014b1783"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "06a4a59bc6f2b0a84131ece231fea92b"
server_key_iv: "6d76a22b831125b03c11ee92c33899e6"
server_key_version: "encryptionkey@0"
I20260927 21:36:12.752895 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 21:36:12.756098 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:12.757261 17758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:36:12.757436 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "94848b97740e407fb3bf19da014b1783"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "06a4a59bc6f2b0a84131ece231fea92b"
server_key_iv: "6d76a22b831125b03c11ee92c33899e6"
server_key_version: "encryptionkey@0"
I20260927 21:36:12.757652 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:12.780814 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:12.784855 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:12.785137 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:12.786010 17758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:12.787262 17758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:12.787366 17758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:12.787513 17758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:12.787586 17758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:12.821808 17758 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:38205
I20260927 21:36:12.821883 17885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:38205 every 8 connection(s)
I20260927 21:36:12.822938 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:36:12.831069 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17758
I20260927 21:36:12.831286 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:36:12.832068 3819 external_mini_cluster.cc:1428] Setting key 2c8e8fb1ecd89a826b1bc6c81bd48301
Sep 27 21:36:12 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544972, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:12.842470 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:12.825435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:45017 (local address 127.3.186.254:44503)
0927 21:36:12.825925 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:12.825932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:12.826990 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 21:36:12.828678 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:12.828682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:12.828747 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:12.828842 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:12.830201 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:12.830936 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:12.833043 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:12.833466 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:12.833692 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:12.837093 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:12.837097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:12.837101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:12.837127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:12.839595 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:12.840231 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:12.840237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:12.840241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:12.840357 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:12.840788 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:12.840795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:12.840799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:12.841078 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:12.841232 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:12.841928 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 21:36:12.842157 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260927 21:36:12.843317 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:12.826017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:45017)
0927 21:36:12.826757 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:12.826809 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:12.828443 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:12.829034 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:12.829041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:12.829492 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:12.830062 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:12.830078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:12.831095 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:12.831101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:12.832915 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:12.832926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:12.833771 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:12.833780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:12.834032 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:12.834701 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:12.834722 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:12.836890 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:12.839727 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:12.839734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:12.839747 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:12.840118 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:12.840483 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:12.840488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:12.840492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:12.840666 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:12.841205 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:12.841212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:12.841509 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 21:36:12.841807 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 21:36:12.842307 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":303,"threads_started":1}
I20260927 21:36:12.844875 17886 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44503
I20260927 21:36:12.845157 17886 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:12.845804 17886 heartbeater.cc:511] Master 127.3.186.254:44503 requested a full tablet report, sending...
I20260927 21:36:12.847685 17697 ts_manager.cc:194] Registered new tserver with Master: 94848b97740e407fb3bf19da014b1783 (127.3.186.193:38205)
I20260927 21:36:12.849050 17697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:45017
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:36:12.878830 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44503
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:13.180894 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:13.181124 17892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:13.181207 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:13.187986 17892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:13.188083 17892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:13.188181 17892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:36:13.197041 17892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:13.198035 17892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:13.199128 17892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:13.211578 17897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:13.211578 17900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:13.211612 17898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:13.212355 17892 server_base.cc:1034] running on GCE node
I20260927 21:36:13.212821 17892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:13.213690 17892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:13.214926 17892 hybrid_clock.cc:648] HybridClock initialized: now 1790544973214849 us; error 65 us; skew 500 ppm
I20260927 21:36:13.215166 17892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:13 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:13.217939 17892 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:36:13.219244 17892 webserver.cc:460] Webserver started at http://127.3.186.194:32945/ using document root <none> and password file <none>
I20260927 21:36:13.219842 17892 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:13.219965 17892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:13.220253 17892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:13.222846 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "4d3601eb86fa4e18b6a135b62c0227de"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "f4393f1e1d3728f874108f0917d70c5a"
server_key_iv: "887770ff644218b3f3469d837eabc1fc"
server_key_version: "encryptionkey@0"
I20260927 21:36:13.223544 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "4d3601eb86fa4e18b6a135b62c0227de"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "f4393f1e1d3728f874108f0917d70c5a"
server_key_iv: "887770ff644218b3f3469d837eabc1fc"
server_key_version: "encryptionkey@0"
I20260927 21:36:13.228276 17892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:36:13.231365 17907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:13.232543 17892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:36:13.232697 17892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "4d3601eb86fa4e18b6a135b62c0227de"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "f4393f1e1d3728f874108f0917d70c5a"
server_key_iv: "887770ff644218b3f3469d837eabc1fc"
server_key_version: "encryptionkey@0"
I20260927 21:36:13.232885 17892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:13.267971 17892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:13.271951 17892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:13.272217 17892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:13.273049 17892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:13.274305 17892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:13.274415 17892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:13.274554 17892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:13.274648 17892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:13.306602 17892 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:42269
I20260927 21:36:13.306690 18019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:42269 every 8 connection(s)
I20260927 21:36:13.307757 17892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:36:13.316859 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 17892
I20260927 21:36:13.317068 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:36:13.317484 3819 external_mini_cluster.cc:1428] Setting key de131534371d02d25e3aa5233dfd2670
Sep 27 21:36:13 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:13.327719 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.310171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:48413 (local address 127.3.186.254:44503)
0927 21:36:13.310368 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:13.310376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:13.311328 (+ 952us) server_negotiation.cc:407] Connection header received
0927 21:36:13.313002 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:13.313008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:13.313098 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:13.313221 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:13.314611 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.315329 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.316774 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.317330 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.317465 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.321149 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:13.321158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:13.321165 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:13.321205 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:13.324660 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:13.325355 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:13.325363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:13.325369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:13.325552 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:13.326045 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:13.326052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:13.326057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:13.326419 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:13.326556 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:13.327249 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 21:36:13.327502 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:36:13.328919 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.310600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:48413)
0927 21:36:13.311188 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:13.311219 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:13.312747 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:13.313432 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:13.313440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:13.313930 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:13.314419 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.314439 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.315479 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.315484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.316631 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.316642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.317559 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.317568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.317787 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.318407 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:13.318427 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:13.320947 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:13.324815 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:13.324821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:13.324830 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:13.325226 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:13.325673 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:13.325677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:13.325680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:13.325932 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:13.326534 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:13.326538 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:13.326812 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 21:36:13.327114 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 21:36:13.327677 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":239,"threads_started":1}
I20260927 21:36:13.330255 18020 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44503
I20260927 21:36:13.330577 18020 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:13.331225 18020 heartbeater.cc:511] Master 127.3.186.254:44503 requested a full tablet report, sending...
I20260927 21:36:13.332513 17697 ts_manager.cc:194] Registered new tserver with Master: 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269)
I20260927 21:36:13.333233 17697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:48413
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:36:13.365061 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44503
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:13.664492 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:13.664739 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:13.664824 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:13.671301 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:13.671388 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:13.671484 18025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:36:13.680130 18025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:13.681237 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:13.682390 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:13.695073 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:13.695181 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:13.695119 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:13.695627 18025 server_base.cc:1034] running on GCE node
I20260927 21:36:13.696339 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:13.697309 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:13.698522 18025 hybrid_clock.cc:648] HybridClock initialized: now 1790544973698465 us; error 71 us; skew 500 ppm
I20260927 21:36:13.698807 18025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:13 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:13.701777 18025 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:36:13.703102 18025 webserver.cc:460] Webserver started at http://127.3.186.195:34073/ using document root <none> and password file <none>
I20260927 21:36:13.703703 18025 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:13.703840 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:13.704120 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:13.706861 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "e943a28179a94947b0dc79afe83dc490"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "18250524d50f7ae28c0d6c45a049aac0"
server_key_iv: "9afe50cbd262e599b4ae476af0d02ef5"
server_key_version: "encryptionkey@0"
I20260927 21:36:13.707612 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "e943a28179a94947b0dc79afe83dc490"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "18250524d50f7ae28c0d6c45a049aac0"
server_key_iv: "9afe50cbd262e599b4ae476af0d02ef5"
server_key_version: "encryptionkey@0"
I20260927 21:36:13.712247 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:36:13.715409 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:13.716585 18025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:36:13.716771 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "e943a28179a94947b0dc79afe83dc490"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "18250524d50f7ae28c0d6c45a049aac0"
server_key_iv: "9afe50cbd262e599b4ae476af0d02ef5"
server_key_version: "encryptionkey@0"
I20260927 21:36:13.716967 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:13.734002 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:13.737823 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:13.738085 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:13.738901 18025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:13.740093 18025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:13.740193 18025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:13.740341 18025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:13.740413 18025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:13.771874 18025 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:40879
I20260927 21:36:13.771950 18152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:40879 every 8 connection(s)
I20260927 21:36:13.772989 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:36:13.781425 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18025
I20260927 21:36:13.781631 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:36:13.782140 3819 external_mini_cluster.cc:1428] Setting key 320f2f0eff2550c8a627466f8a6380ea
Sep 27 21:36:13 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:13.793581 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.775342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:57483 (local address 127.3.186.254:44503)
0927 21:36:13.775564 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:13.775570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:13.776588 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 21:36:13.778236 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:13.778240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:13.778306 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:13.778390 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:13.779663 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.780358 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.782133 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.782839 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.782953 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.787483 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:13.787491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:13.787495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:13.787533 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:13.790945 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:13.791768 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:13.791772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:13.791774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:13.791882 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:13.792241 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:13.792244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:13.792246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:13.792484 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:13.792585 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:13.793167 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 21:36:13.793305 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:36:13.794378 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.775758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:57483)
0927 21:36:13.776432 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:13.776476 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:13.778015 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:13.778588 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:13.778595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:13.779036 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:13.779526 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.779542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.780549 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.780554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.781904 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.781917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.783294 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.783302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.783580 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.784264 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:13.784300 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:13.787300 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:13.791092 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:13.791100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:13.791129 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:13.791652 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:13.791984 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:13.791987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:13.791989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:13.792144 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:13.792583 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:13.792588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:13.792841 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 21:36:13.793083 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 21:36:13.793384 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":284,"threads_started":1}
I20260927 21:36:13.795606 18153 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44503
I20260927 21:36:13.795894 18153 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:13.796695 18153 heartbeater.cc:511] Master 127.3.186.254:44503 requested a full tablet report, sending...
I20260927 21:36:13.798151 17697 ts_manager.cc:194] Registered new tserver with Master: e943a28179a94947b0dc79afe83dc490 (127.3.186.195:40879)
I20260927 21:36:13.798887 17697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:57483
I20260927 21:36:13.809732 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:36:13.826363 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.814103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.3.186.254:44503)
0927 21:36:13.814276 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:13.814282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:13.814716 (+ 434us) server_negotiation.cc:407] Connection header received
0927 21:36:13.814905 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:13.814911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:13.815002 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:13.815105 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:13.816408 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.817077 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.818371 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.818778 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.818912 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.821329 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:13.821340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:13.821343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:13.821370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:13.823782 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:13.824391 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:13.824398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:13.824401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:13.824531 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:13.824972 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:13.824977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:13.824983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:13.825272 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:13.825550 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:13.825884 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 21:36:13.826077 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:36:13.829788 17697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:36:13.836246 17697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:36:13.851750 17886 heartbeater.cc:503] Master 127.3.186.254:44503 was elected leader, sending a full tablet report...
I20260927 21:36:13.868022 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.846337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:40879 (local address 127.0.0.1:35872)
0927 21:36:13.856165 (+ 9828us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:13.856219 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:13.856395 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:13.859112 (+ 2717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:13.859118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:13.859147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:13.859595 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.859611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.862608 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.862613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.865051 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.865063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.866865 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.866872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.867208 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.867275 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 21:36:13.867400 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 21:36:13.867588 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9541,"mutex_wait_us":47,"thread_start_us":12229,"threads_started":1}
I20260927 21:36:13.868645 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.846720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.3.186.193:38205)
0927 21:36:13.861163 (+ 14443us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:13.861168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:13.861189 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:36:13.861255 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:13.861259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:13.861394 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:13.862657 (+ 1263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:13.862931 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.863948 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.866096 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.867163 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.867297 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.867513 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:13.867738 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:36:13.867892 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14329,"thread_start_us":14863,"threads_started":1}
I20260927 21:36:13.868826 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.846553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.3.186.195:40879)
0927 21:36:13.858598 (+ 12045us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:13.858605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:13.858636 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:36:13.858722 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:13.858729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:13.858915 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:13.859105 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:13.859816 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.861770 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.865260 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.866667 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.866791 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.867045 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:13.868336 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0927 21:36:13.868506 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11856,"thread_start_us":11681,"threads_started":1}
I20260927 21:36:13.868182 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.846547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:47992)
0927 21:36:13.858879 (+ 12332us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:13.858908 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:13.859039 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:13.861951 (+ 2912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:13.861968 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:13.861991 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:13.862426 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.862440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.864078 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.864085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.865953 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.865965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.867389 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.867397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.867555 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.867588 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:36:13.867678 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:36:13.867805 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12182,"thread_start_us":15070,"threads_started":1}
I20260927 21:36:13.869910 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.847671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:42269 (local address 127.0.0.1:48748)
0927 21:36:13.861411 (+ 13740us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:13.861432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:13.861557 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:13.862327 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:13.862336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:13.862359 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:13.863045 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.863060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.864989 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.864995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.867211 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:13.867222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.869108 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.869115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:13.869318 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.869351 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:36:13.869449 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:36:13.869586 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13622,"mutex_wait_us":51,"thread_start_us":8047,"threads_started":1}
I20260927 21:36:13.869979 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:13.848243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.3.186.194:42269)
0927 21:36:13.855644 (+ 7401us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:13.855656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:13.861598 (+ 5942us) server_negotiation.cc:407] Connection header received
0927 21:36:13.861696 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:13.861703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:13.861918 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:13.862093 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:13.863241 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.864815 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.867423 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:13.868876 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:13.869007 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:13.869185 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:13.869537 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 21:36:13.869688 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7188,"thread_start_us":7067,"threads_started":1}
I20260927 21:36:13.872426 18088 tablet_service.cc:1467] Processing CreateTablet for tablet fc3e35d5e96e452da5cbf33ec9b1708c (DEFAULT_TABLE table=test-table [id=9868fda3fb594d03833fc4fc9c2a2d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:13.872574 17821 tablet_service.cc:1467] Processing CreateTablet for tablet fc3e35d5e96e452da5cbf33ec9b1708c (DEFAULT_TABLE table=test-table [id=9868fda3fb594d03833fc4fc9c2a2d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:13.873200 17955 tablet_service.cc:1467] Processing CreateTablet for tablet fc3e35d5e96e452da5cbf33ec9b1708c (DEFAULT_TABLE table=test-table [id=9868fda3fb594d03833fc4fc9c2a2d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:13.873885 17821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3e35d5e96e452da5cbf33ec9b1708c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:13.873895 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3e35d5e96e452da5cbf33ec9b1708c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:13.874405 17955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3e35d5e96e452da5cbf33ec9b1708c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:13.884291 18171 tablet_bootstrap.cc:492] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Bootstrap starting.
I20260927 21:36:13.887435 18171 tablet_bootstrap.cc:654] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:13.887454 18172 tablet_bootstrap.cc:492] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Bootstrap starting.
I20260927 21:36:13.888732 18171 log.cc:824] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:13.888906 18173 tablet_bootstrap.cc:492] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Bootstrap starting.
I20260927 21:36:13.890913 18172 tablet_bootstrap.cc:654] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:13.891786 18173 tablet_bootstrap.cc:654] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:13.892398 18172 log.cc:824] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:13.892710 18173 log.cc:824] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:13.895032 18172 tablet_bootstrap.cc:492] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: No bootstrap required, opened a new log
I20260927 21:36:13.895129 18171 tablet_bootstrap.cc:492] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: No bootstrap required, opened a new log
I20260927 21:36:13.895329 18172 ts_tablet_manager.cc:1397] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:36:13.895412 18171 ts_tablet_manager.cc:1397] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260927 21:36:13.896914 18173 tablet_bootstrap.cc:492] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: No bootstrap required, opened a new log
I20260927 21:36:13.897136 18173 ts_tablet_manager.cc:1397] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 21:36:13.902186 18172 raft_consensus.cc:348] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:13.902530 18172 raft_consensus.cc:374] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:13.902654 18172 raft_consensus.cc:729] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3601eb86fa4e18b6a135b62c0227de, State: Initialized, Role: FOLLOWER
I20260927 21:36:13.903285 18172 consensus_queue.cc:260] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:13.904376 18020 heartbeater.cc:503] Master 127.3.186.254:44503 was elected leader, sending a full tablet report...
I20260927 21:36:13.904865 18172 ts_tablet_manager.cc:1428] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Time spent starting tablet: real 0.009s user 0.005s sys 0.006s
I20260927 21:36:13.907191 18171 raft_consensus.cc:348] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:13.907191 18173 raft_consensus.cc:348] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:13.907573 18173 raft_consensus.cc:374] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:13.907575 18171 raft_consensus.cc:374] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:13.907675 18171 raft_consensus.cc:729] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e943a28179a94947b0dc79afe83dc490, State: Initialized, Role: FOLLOWER
I20260927 21:36:13.907670 18173 raft_consensus.cc:729] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94848b97740e407fb3bf19da014b1783, State: Initialized, Role: FOLLOWER
I20260927 21:36:13.908342 18171 consensus_queue.cc:260] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:13.908342 18173 consensus_queue.cc:260] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:13.914587 18153 heartbeater.cc:503] Master 127.3.186.254:44503 was elected leader, sending a full tablet report...
I20260927 21:36:13.914862 18173 ts_tablet_manager.cc:1428] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260927 21:36:13.915294 18171 ts_tablet_manager.cc:1428] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
W20260927 21:36:14.024485 18154 tablet.cc:2369] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:36:14.059540 18021 tablet.cc:2369] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:36:14.075105 17887 tablet.cc:2369] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:36:14.142187 18179 raft_consensus.cc:482] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:36:14.142483 18179 raft_consensus.cc:504] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:14.144141 18179 leader_election.cc:288] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94848b97740e407fb3bf19da014b1783 (127.3.186.193:38205), 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269)
I20260927 21:36:14.150344 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.144621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:40057 (local address 127.3.186.193:38205)
0927 21:36:14.144850 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:14.144858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:14.145023 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:36:14.145249 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:14.145256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:14.145339 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:14.145443 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:14.145847 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.146543 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.148303 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.149538 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.149671 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.149722 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:14.150076 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:36:14.150175 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:36:14.150420 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.144501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.3.186.195:40057)
0927 21:36:14.144906 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:14.144925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:14.145097 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:14.145469 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:14.145474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:14.145501 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:14.145711 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:14.145720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.146671 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.146679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:14.148128 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:14.148145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.149681 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.149687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:14.149876 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.149915 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:36:14.150019 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:36:14.150144 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3}
I20260927 21:36:14.151533 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.144679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:42269 (local address 127.3.186.195:37873)
0927 21:36:14.145439 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:14.145453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:14.145645 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:14.145986 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:14.145991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:14.146014 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:14.146326 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:14.146335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.147325 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.147331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:14.149315 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:14.149329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.150994 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.150999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:14.151108 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.151129 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:36:14.151207 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:36:14.151285 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":177,"thread_start_us":422,"threads_started":1}
I20260927 21:36:14.151612 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.144694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:37873 (local address 127.3.186.194:42269)
0927 21:36:14.144897 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:14.144926 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:14.145556 (+ 630us) server_negotiation.cc:407] Connection header received
0927 21:36:14.145794 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:14.145800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:14.145868 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:14.145962 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:14.146485 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.147181 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.149491 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.150771 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.150874 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.150972 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:14.151253 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 21:36:14.151378 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:36:14.151441 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3e35d5e96e452da5cbf33ec9b1708c" candidate_uuid: "e943a28179a94947b0dc79afe83dc490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94848b97740e407fb3bf19da014b1783" is_pre_election: true
I20260927 21:36:14.151979 17841 raft_consensus.cc:2454] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e943a28179a94947b0dc79afe83dc490 in term 0.
I20260927 21:36:14.152607 18042 leader_election.cc:302] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94848b97740e407fb3bf19da014b1783, e943a28179a94947b0dc79afe83dc490; no voters:
I20260927 21:36:14.152602 17975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3e35d5e96e452da5cbf33ec9b1708c" candidate_uuid: "e943a28179a94947b0dc79afe83dc490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3601eb86fa4e18b6a135b62c0227de" is_pre_election: true
I20260927 21:36:14.152966 18179 raft_consensus.cc:2781] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:36:14.153083 18179 raft_consensus.cc:482] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:36:14.153218 18179 raft_consensus.cc:3037] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:14.153190 17975 raft_consensus.cc:2454] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e943a28179a94947b0dc79afe83dc490 in term 0.
I20260927 21:36:14.155393 18179 raft_consensus.cc:504] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:14.156038 18179 leader_election.cc:288] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [CANDIDATE]: Term 1 election: Requested vote from peers 94848b97740e407fb3bf19da014b1783 (127.3.186.193:38205), 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269)
I20260927 21:36:14.156368 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3e35d5e96e452da5cbf33ec9b1708c" candidate_uuid: "e943a28179a94947b0dc79afe83dc490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94848b97740e407fb3bf19da014b1783"
I20260927 21:36:14.156517 17975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3e35d5e96e452da5cbf33ec9b1708c" candidate_uuid: "e943a28179a94947b0dc79afe83dc490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3601eb86fa4e18b6a135b62c0227de"
I20260927 21:36:14.156563 17841 raft_consensus.cc:3037] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:14.156697 17975 raft_consensus.cc:3037] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:14.159541 17841 raft_consensus.cc:2454] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e943a28179a94947b0dc79afe83dc490 in term 1.
I20260927 21:36:14.159578 17975 raft_consensus.cc:2454] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e943a28179a94947b0dc79afe83dc490 in term 1.
I20260927 21:36:14.159884 18042 leader_election.cc:302] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94848b97740e407fb3bf19da014b1783, e943a28179a94947b0dc79afe83dc490; no voters:
I20260927 21:36:14.160185 18179 raft_consensus.cc:2781] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:14.160578 18179 raft_consensus.cc:686] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 1 LEADER]: Becoming Leader. State: Replica: e943a28179a94947b0dc79afe83dc490, State: Running, Role: LEADER
I20260927 21:36:14.161072 18179 consensus_queue.cc:237] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:14.165149 17696 catalog_manager.cc:5495] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 reported cstate change: term changed from 0 to 1, leader changed from <none> to e943a28179a94947b0dc79afe83dc490 (127.3.186.195). New cstate: current_term: 1 leader_uuid: "e943a28179a94947b0dc79afe83dc490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } health_report { overall_health: UNKNOWN } } }
I20260927 21:36:14.230233 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.225285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.3.186.195:40879)
0927 21:36:14.225440 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:14.225444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:14.225461 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:36:14.225690 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:14.225695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:14.225814 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:14.226006 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:36:14.226302 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.227159 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.228506 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.228847 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.228954 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.229327 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:36:14.229782 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:36:14.229879 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:14.230015 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:36:14.230069 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:36:14.237550 17841 raft_consensus.cc:1261] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Refusing update from remote peer e943a28179a94947b0dc79afe83dc490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:14.237547 17975 raft_consensus.cc:1261] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Refusing update from remote peer e943a28179a94947b0dc79afe83dc490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:14.238603 18181 consensus_queue.cc:1035] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:14.238965 18179 consensus_queue.cc:1035] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:14.252616 18186 mvcc.cc:205] Tried to move back new op lower bound from 7334072214467313664 to 7334072214167576576. Current Snapshot: MvccSnapshot[applied={T|T < 7334072214467313664}]
I20260927 21:36:14.264504 17695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:14.264844 17695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:14.268805 17695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Sending DeleteTablet for 3 replicas of tablet fc3e35d5e96e452da5cbf33ec9b1708c
I20260927 21:36:14.269758 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3e35d5e96e452da5cbf33ec9b1708c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:14 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:47992
I20260927 21:36:14.270323 18088 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3e35d5e96e452da5cbf33ec9b1708c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:14 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:35872
I20260927 21:36:14.271130 18194 tablet_replica.cc:330] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: stopping tablet replica
I20260927 21:36:14.271555 18194 raft_consensus.cc:2229] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:36:14.271757 17955 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3e35d5e96e452da5cbf33ec9b1708c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:14 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:48748
I20260927 21:36:14.272253 18194 raft_consensus.cc:2258] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:14.274149 18194 ts_tablet_manager.cc:1905] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:14.279336 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17758
I20260927 21:36:14.281212 18194 ts_tablet_manager.cc:1918] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:14.281446 18194 log.cc:1196] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/fc3e35d5e96e452da5cbf33ec9b1708c
I20260927 21:36:14.282037 18194 ts_tablet_manager.cc:1939] T fc3e35d5e96e452da5cbf33ec9b1708c P e943a28179a94947b0dc79afe83dc490: Deleting consensus metadata
I20260927 21:36:14.283634 17681 catalog_manager.cc:4841] TS e943a28179a94947b0dc79afe83dc490 (127.3.186.195:40879): tablet fc3e35d5e96e452da5cbf33ec9b1708c (table test-table [id=9868fda3fb594d03833fc4fc9c2a2d15]) successfully deleted
I20260927 21:36:14.286635 18195 tablet_replica.cc:330] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: stopping tablet replica
I20260927 21:36:14.287093 18195 raft_consensus.cc:2229] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:14.287639 18193 tablet_replica.cc:330] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: stopping tablet replica
I20260927 21:36:14.288106 18195 raft_consensus.cc:2258] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:14.288550 18193 raft_consensus.cc:2229] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:14.290727 18193 raft_consensus.cc:2258] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:14.291666 18195 ts_tablet_manager.cc:1905] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:14.292927 18193 ts_tablet_manager.cc:1905] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:36:14 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:36:14.298257 18195 ts_tablet_manager.cc:1918] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:14.298485 18195 log.cc:1196] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/fc3e35d5e96e452da5cbf33ec9b1708c
I20260927 21:36:14.299006 18195 ts_tablet_manager.cc:1939] T fc3e35d5e96e452da5cbf33ec9b1708c P 4d3601eb86fa4e18b6a135b62c0227de: Deleting consensus metadata
I20260927 21:36:14.299897 18193 ts_tablet_manager.cc:1918] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:14.300067 18193 log.cc:1196] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/fc3e35d5e96e452da5cbf33ec9b1708c
I20260927 21:36:14.300685 17683 catalog_manager.cc:4841] TS 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269): tablet fc3e35d5e96e452da5cbf33ec9b1708c (table test-table [id=9868fda3fb594d03833fc4fc9c2a2d15]) successfully deleted
I20260927 21:36:14.300748 18193 ts_tablet_manager.cc:1939] T fc3e35d5e96e452da5cbf33ec9b1708c P 94848b97740e407fb3bf19da014b1783: Deleting consensus metadata
I20260927 21:36:14.302295 17682 catalog_manager.cc:4841] TS 94848b97740e407fb3bf19da014b1783 (127.3.186.193:38205): tablet fc3e35d5e96e452da5cbf33ec9b1708c (table test-table [id=9868fda3fb594d03833fc4fc9c2a2d15]) successfully deleted
I20260927 21:36:14.305606 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.280386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.3.186.193:38205)
0927 21:36:14.280536 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:14.280542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:14.286047 (+ 5505us) server_negotiation.cc:407] Connection header received
0927 21:36:14.286236 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:14.286240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:14.286313 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:14.286413 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:14.288396 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.289502 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.292243 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.292895 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.293003 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.298668 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:14.298684 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:14.298700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:14.298740 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:14.302570 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:14.303326 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:14.303332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:14.303334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:14.303459 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:14.303934 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:14.303957 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:14.303960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:14.304303 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:14.304509 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:14.304997 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 21:36:14.305190 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:36:14.306274 17881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:14.404168 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17758
I20260927 21:36:14.422259 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17892
Sep 27 21:36:14 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:36:14.439615 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.423096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.3.186.194:42269)
0927 21:36:14.423245 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:14.423249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:14.423265 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:36:14.423453 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:14.423456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:14.423517 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:14.423597 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:14.425246 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.426043 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.428213 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.428626 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.428720 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.433581 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:14.433595 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:14.433609 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:14.433644 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:14.436672 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:14.437418 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:14.437426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:14.437428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:14.437703 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:14.438091 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:14.438097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:14.438101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:14.438467 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:14.438755 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:14.439068 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 21:36:14.439280 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:36:14.440380 18015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:14.527179 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17892
I20260927 21:36:14.544587 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18025
Sep 27 21:36:14 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:36:14.563122 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:14.545450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.3.186.195:40879)
0927 21:36:14.545689 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:14.545695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:14.545719 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:36:14.545793 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:14.545799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:14.545900 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:14.546012 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:14.547562 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.548760 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.550493 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:14.551189 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:14.551330 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:14.556002 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:14.556017 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:14.556037 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:14.556078 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:14.560172 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:14.560827 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:14.560836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:14.560840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:14.561032 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:14.561400 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:14.561405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:14.561408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:14.561909 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:14.562218 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:14.562475 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 21:36:14.562685 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 21:36:14.563812 18148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:14.681808 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18025
I20260927 21:36:14.697335 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 17665
I20260927 21:36:14.698397 17727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:14.804904 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 17665
I20260927 21:36:14.831259 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44503
--webserver_interface=127.3.186.254
--webserver_port=40311
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:15.145056 18201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:15.145357 18201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:15.145546 18201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:15.152274 18201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:15.152398 18201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:15.152479 18201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:15.152557 18201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:15.152621 18201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:15.161405 18201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=40311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:15.162482 18201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:15.163627 18201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:15.184580 18207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:15.184602 18210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:15.184634 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:15.185477 18201 server_base.cc:1034] running on GCE node
I20260927 21:36:15.186308 18201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:15.187721 18201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:15.188963 18201 hybrid_clock.cc:648] HybridClock initialized: now 1790544975188900 us; error 47 us; skew 500 ppm
I20260927 21:36:15.189327 18201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:15 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544975, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:15.192670 18201 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:15.194571 18201 webserver.cc:460] Webserver started at http://127.3.186.254:40311/ using document root <none> and password file <none>
I20260927 21:36:15.195314 18201 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:15.195480 18201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:15.201061 18201 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 21:36:15.203846 18217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:15.205093 18201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:36:15.205291 18201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "2b409676f4b74d888c62f4516747ece3"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "fcd24e7b09abe48e0a98e0d3b82dd2be"
server_key_iv: "0fcd104b7a2e37aa75c68b470a99414b"
server_key_version: "encryptionkey@0"
I20260927 21:36:15.206382 18201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:15.226052 18201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:15.230039 18201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:15.230283 18201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:15.247860 18201 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44503
I20260927 21:36:15.247890 18268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44503 every 8 connection(s)
I20260927 21:36:15.249168 18201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:15.254998 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18201
I20260927 21:36:15.255587 18269 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:36:15.256009 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:38205
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:44503
--webserver_port=43987
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:15.260380 18269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap starting.
I20260927 21:36:15.279464 18269 log.cc:824] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:15.290624 18269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:36:15.291102 18269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap complete.
I20260927 21:36:15.299517 18269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:15.300375 18269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b409676f4b74d888c62f4516747ece3, State: Initialized, Role: FOLLOWER
I20260927 21:36:15.300925 18269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:15.301155 18269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:15.301288 18269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:15.301461 18269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:36:15.305070 18269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:15.305608 18269 leader_election.cc:302] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b409676f4b74d888c62f4516747ece3; no voters:
I20260927 21:36:15.306205 18269 leader_election.cc:288] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:36:15.306330 18273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:36:15.307526 18273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 LEADER]: Becoming Leader. State: Replica: 2b409676f4b74d888c62f4516747ece3, State: Running, Role: LEADER
I20260927 21:36:15.308094 18273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:15.308411 18269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:15.310591 18275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b409676f4b74d888c62f4516747ece3. Latest consensus state: current_term: 2 leader_uuid: "2b409676f4b74d888c62f4516747ece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } } }
I20260927 21:36:15.311033 18275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:15.311612 18274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b409676f4b74d888c62f4516747ece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } } }
I20260927 21:36:15.311885 18274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:15.311946 18279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:15.322324 18279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:15.324054 18279 catalog_manager.cc:1259] Loaded cluster ID: f90f3c0816e5425eb7eb41c08c680de4
I20260927 21:36:15.324218 18279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:15.327982 18279 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 21:36:15.352394 18279 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 21:36:15.362645 18279 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:15.364106 18279 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Loaded TSK: 0
I20260927 21:36:15.365521 18279 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:36:15.588708 18271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:15.588979 18271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:15.589144 18271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:15.596068 18271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:15.596189 18271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:15.596355 18271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:36:15.605273 18271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:38205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=43987
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:15.606433 18271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:15.607764 18271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:15.621757 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:15.621730 18298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:15.622637 18271 server_base.cc:1034] running on GCE node
W20260927 21:36:15.623205 18300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:15.623852 18271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:15.624708 18271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:15.625933 18271 hybrid_clock.cc:648] HybridClock initialized: now 1790544975625877 us; error 56 us; skew 500 ppm
I20260927 21:36:15.626233 18271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:15 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544975, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:15.629982 18271 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:36:15.631551 18271 webserver.cc:460] Webserver started at http://127.3.186.193:43987/ using document root <none> and password file <none>
I20260927 21:36:15.632345 18271 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:15.632503 18271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:15.639048 18271 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:36:15.642095 18307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:15.643672 18271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:36:15.643823 18271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "94848b97740e407fb3bf19da014b1783"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "06a4a59bc6f2b0a84131ece231fea92b"
server_key_iv: "6d76a22b831125b03c11ee92c33899e6"
server_key_version: "encryptionkey@0"
I20260927 21:36:15.645110 18271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:15.696163 18271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:15.700322 18271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:15.700615 18271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:15.701668 18271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:15.703058 18271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:15.703213 18271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:15.703316 18271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:15.703408 18271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:15.735774 18271 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:38205
I20260927 21:36:15.735929 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:38205 every 8 connection(s)
I20260927 21:36:15.737135 18271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:36:15.739209 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18271
I20260927 21:36:15.739884 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:42269
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:44503
--webserver_port=32945
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:36:15 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544975, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:15.760676 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:15.740059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:50615 (local address 127.3.186.254:44503)
0927 21:36:15.740674 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:15.740690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:15.741241 (+ 551us) server_negotiation.cc:407] Connection header received
0927 21:36:15.743315 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:15.743335 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:15.743776 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:15.744166 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:15.745492 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:15.747141 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:15.748943 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:15.750083 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:15.750276 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:15.753877 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:15.753884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:15.753896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:15.753931 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:15.756680 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:15.757619 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:15.757624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:15.757629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:15.757768 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:15.758244 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:15.758246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:15.758248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:15.758632 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:15.758770 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:15.759626 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 21:36:15.759898 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":229,"threads_started":1}
I20260927 21:36:15.761210 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:15.740212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:50615)
0927 21:36:15.741058 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:15.741098 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:15.742745 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:15.744274 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:15.744280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:15.744760 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:15.745261 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:15.745276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:15.747305 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:15.747312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:15.748817 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:15.748829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:15.750272 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:15.750277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:15.750468 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:15.751098 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:15.751117 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:15.753638 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:15.756818 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:15.756829 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:15.756844 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:15.757478 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:15.757878 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:15.757883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:15.757888 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:15.758131 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:15.758762 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:15.758772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:15.759184 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 21:36:15.759476 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 21:36:15.759976 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":357,"threads_started":1}
I20260927 21:36:15.763706 18420 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44503
I20260927 21:36:15.763978 18420 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:15.764595 18420 heartbeater.cc:511] Master 127.3.186.254:44503 requested a full tablet report, sending...
I20260927 21:36:15.766592 18234 ts_manager.cc:194] Registered new tserver with Master: 94848b97740e407fb3bf19da014b1783 (127.3.186.193:38205)
I20260927 21:36:15.767943 18234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:50615
W20260927 21:36:16.047391 18424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:16.047616 18424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:16.047761 18424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:16.055152 18424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:16.055241 18424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:16.055446 18424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:36:16.064214 18424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:42269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=32945
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:16.065307 18424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:16.066601 18424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:16.078650 18429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:16.078685 18432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:16.078611 18430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:16.079278 18424 server_base.cc:1034] running on GCE node
I20260927 21:36:16.079829 18424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:16.080650 18424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:16.081853 18424 hybrid_clock.cc:648] HybridClock initialized: now 1790544976081805 us; error 45 us; skew 500 ppm
I20260927 21:36:16.082126 18424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:16 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544976, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:16.084846 18424 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:36:16.086202 18424 webserver.cc:460] Webserver started at http://127.3.186.194:32945/ using document root <none> and password file <none>
I20260927 21:36:16.086858 18424 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:16.087023 18424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:16.091598 18424 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:36:16.093966 18439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:16.095110 18424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 21:36:16.095273 18424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "4d3601eb86fa4e18b6a135b62c0227de"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "f4393f1e1d3728f874108f0917d70c5a"
server_key_iv: "887770ff644218b3f3469d837eabc1fc"
server_key_version: "encryptionkey@0"
I20260927 21:36:16.096149 18424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:16.110774 18424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:16.114205 18424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:16.114531 18424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:16.115419 18424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:16.116545 18424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:16.116622 18424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:16.116693 18424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:16.116746 18424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:16.147904 18424 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:42269
I20260927 21:36:16.147948 18551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:42269 every 8 connection(s)
I20260927 21:36:16.149262 18424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:36:16.151809 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18424
I20260927 21:36:16.152400 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:40879
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:44503
--webserver_port=34073
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:36:16 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544976, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:16.168998 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.151578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:42393 (local address 127.3.186.254:44503)
0927 21:36:16.151792 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.151800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.152959 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 21:36:16.154830 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.154838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.154938 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.155048 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:16.156406 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.157457 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.159010 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.159414 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.159538 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.163469 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:16.163474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:16.163477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:16.163508 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:16.166021 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:16.166868 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:16.166881 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:16.166883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:16.166991 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:16.167444 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:16.167447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:16.167449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:16.167659 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:16.167757 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.168596 (+ 839us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.168759 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:36:16.170379 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.152042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:42393)
0927 21:36:16.152787 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.152827 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.154538 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.155273 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.155281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.155729 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:16.156233 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.156248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.157648 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.157652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.158857 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.158869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.159611 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.159617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.159919 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.160536 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:16.160566 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:16.163271 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:16.166152 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:16.166163 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:16.166178 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:16.166752 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:16.167093 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:16.167100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:16.167107 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:16.167334 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:16.167780 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:16.167790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:16.168176 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.168465 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.169002 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":314,"threads_started":1}
I20260927 21:36:16.171749 18552 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44503
I20260927 21:36:16.172021 18552 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:16.172643 18552 heartbeater.cc:511] Master 127.3.186.254:44503 requested a full tablet report, sending...
I20260927 21:36:16.173995 18234 ts_manager.cc:194] Registered new tserver with Master: 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269)
I20260927 21:36:16.174757 18234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:42393
W20260927 21:36:16.456347 18555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:16.456606 18555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:16.456768 18555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:16.463518 18555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:16.463640 18555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:16.463805 18555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:36:16.472635 18555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:40879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=34073
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:16.473809 18555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:16.474958 18555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:16.486824 18560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:16.486775 18561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:16.486939 18563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:16.487931 18555 server_base.cc:1034] running on GCE node
I20260927 21:36:16.488494 18555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:16.489338 18555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:16.490567 18555 hybrid_clock.cc:648] HybridClock initialized: now 1790544976490534 us; error 48 us; skew 500 ppm
I20260927 21:36:16.490911 18555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:16 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544976, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:16.493770 18555 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:36:16.495108 18555 webserver.cc:460] Webserver started at http://127.3.186.195:34073/ using document root <none> and password file <none>
I20260927 21:36:16.495718 18555 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:16.495885 18555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:16.500649 18555 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:36:16.503152 18570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:16.504369 18555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:36:16.504532 18555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "e943a28179a94947b0dc79afe83dc490"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "18250524d50f7ae28c0d6c45a049aac0"
server_key_iv: "9afe50cbd262e599b4ae476af0d02ef5"
server_key_version: "encryptionkey@0"
I20260927 21:36:16.505419 18555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:16.543771 18555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:16.547273 18555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:16.547515 18555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:16.548389 18555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:16.549585 18555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:16.549722 18555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:16.549820 18555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:16.549866 18555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:16.580538 18555 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:40879
I20260927 21:36:16.580627 18682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:40879 every 8 connection(s)
I20260927 21:36:16.581910 18555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:36:16.584364 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18555
Sep 27 21:36:16 dist-test-slave-pqr8 krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544976, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:16.604663 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.585257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.3.186.254:44503)
0927 21:36:16.589389 (+ 4132us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.589394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.589941 (+ 547us) server_negotiation.cc:407] Connection header received
0927 21:36:16.590086 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.590090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.590139 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.590218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:16.591951 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.592748 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.594421 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.595002 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.595095 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.598055 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:16.598061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:16.598065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:16.598099 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:16.601398 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:16.602371 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:16.602378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:16.602382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:16.602533 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:16.603115 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:16.603119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:16.603121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:16.603383 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:16.603669 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.603964 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.604192 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":4020,"thread_start_us":3914,"threads_started":1}
I20260927 21:36:16.604780 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.584113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:55019 (local address 127.3.186.254:44503)
0927 21:36:16.584324 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.584331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.585539 (+ 1208us) server_negotiation.cc:407] Connection header received
0927 21:36:16.589017 (+ 3478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.589026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.589144 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.589276 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:16.591244 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.592047 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.593992 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.594463 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.594604 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.598353 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:16.598359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:16.598363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:16.598402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:16.601326 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:16.602129 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:16.602138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:16.602143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:16.602291 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:16.602803 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:16.602808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:16.602812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:16.603238 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:16.603421 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.604069 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.604346 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":74}
I20260927 21:36:16.605759 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.584571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:55019)
0927 21:36:16.585334 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.585381 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.588623 (+ 3242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.589586 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.589598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.590280 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:16.591064 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.591094 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.592200 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.592204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.593851 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.593867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.594676 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.594681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.594864 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.595543 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:16.595578 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:16.598135 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:16.601463 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:16.601474 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:16.601489 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:16.601982 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:16.602413 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:16.602429 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:16.602432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:16.602651 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:16.603392 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:16.603399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:16.603710 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.603954 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.604419 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260927 21:36:16.607146 18683 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44503
I20260927 21:36:16.607442 18683 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:16.608198 18683 heartbeater.cc:511] Master 127.3.186.254:44503 requested a full tablet report, sending...
I20260927 21:36:16.609655 18233 ts_manager.cc:194] Registered new tserver with Master: e943a28179a94947b0dc79afe83dc490 (127.3.186.195:40879)
I20260927 21:36:16.610572 18233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:55019
I20260927 21:36:16.619342 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:36:16.625427 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.620222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.3.186.254:44503)
0927 21:36:16.620402 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.620407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.620680 (+ 273us) server_negotiation.cc:407] Connection header received
0927 21:36:16.620874 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.620877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.620974 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.621103 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:36:16.621456 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.622154 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.623528 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.623916 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.624039 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.624514 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:36:16.624912 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:36:16.625063 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.625139 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.625209 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:36:16.626250 18233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:36:16.632946 18233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:36:16.666523 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.642968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43728)
0927 21:36:16.655885 (+ 12917us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.655940 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.656128 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.657370 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.657376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.657411 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:16.658075 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.658090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.660338 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.660346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.662666 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.662673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.665653 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.665661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.665853 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.665918 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.666054 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.666220 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12677,"thread_start_us":18335,"threads_started":1}
I20260927 21:36:16.667321 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.643205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.3.186.193:38205)
0927 21:36:16.655331 (+ 12126us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.655338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.656511 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 21:36:16.656622 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.656631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.656903 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.657092 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:16.658315 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.660186 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.664234 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.665368 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.665501 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.666744 (+ 1243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.666820 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.667009 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11947,"thread_start_us":13055,"threads_started":1}
I20260927 21:36:16.668663 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.643256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:40879 (local address 127.0.0.1:45090)
0927 21:36:16.661636 (+ 18380us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.661655 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.661767 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.662301 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.662306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.662327 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:16.662659 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.662671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.664211 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.664217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.666083 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.666094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.667916 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.667922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.668106 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.668155 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.668241 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.668350 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18194,"thread_start_us":12267,"threads_started":1}
I20260927 21:36:16.668754 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.643529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.3.186.195:40879)
0927 21:36:16.661040 (+ 17511us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.661048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.661759 (+ 711us) server_negotiation.cc:407] Connection header received
0927 21:36:16.661888 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.661894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.662122 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.662298 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:16.662828 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.664070 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.666255 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.667721 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.667843 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.668043 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.668309 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.668463 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17368,"thread_start_us":17510,"threads_started":1}
I20260927 21:36:16.669925 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.644889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:42269 (local address 127.0.0.1:36468)
0927 21:36:16.661803 (+ 16914us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.661820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.662193 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.662998 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.663002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.663026 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:16.663388 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.663399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.664913 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.664918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.667180 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.667191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.669148 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.669158 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.669321 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.669351 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.669444 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.669571 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16786,"mutex_wait_us":1,"thread_start_us":17018,"threads_started":1}
I20260927 21:36:16.670763 18355 tablet_service.cc:1467] Processing CreateTablet for tablet 07d607de5ea34a479f0b3c6eb404839f (DEFAULT_TABLE table=test-table [id=62e9a8483fa04af99c6cc5df3dd069ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:16.671865 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d607de5ea34a479f0b3c6eb404839f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:16.671579 18618 tablet_service.cc:1467] Processing CreateTablet for tablet 07d607de5ea34a479f0b3c6eb404839f (DEFAULT_TABLE table=test-table [id=62e9a8483fa04af99c6cc5df3dd069ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:16.672273 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.645626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.3.186.194:42269)
0927 21:36:16.651999 (+ 6373us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.652010 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.661901 (+ 9891us) server_negotiation.cc:407] Connection header received
0927 21:36:16.662375 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.662385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.662587 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.662694 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:16.663548 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.664772 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.667320 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.668686 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.668808 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.669028 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.669531 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.669694 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6188,"thread_start_us":6076,"threads_started":1}
I20260927 21:36:16.672772 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d607de5ea34a479f0b3c6eb404839f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:16.675305 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 07d607de5ea34a479f0b3c6eb404839f (DEFAULT_TABLE table=test-table [id=62e9a8483fa04af99c6cc5df3dd069ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:16.676452 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d607de5ea34a479f0b3c6eb404839f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:16.683320 18699 tablet_bootstrap.cc:492] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Bootstrap starting.
I20260927 21:36:16.686295 18699 tablet_bootstrap.cc:654] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:16.687041 18700 tablet_bootstrap.cc:492] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Bootstrap starting.
I20260927 21:36:16.687683 18699 log.cc:824] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:16.689803 18702 tablet_bootstrap.cc:492] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Bootstrap starting.
I20260927 21:36:16.690017 18700 tablet_bootstrap.cc:654] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:16.690675 18699 tablet_bootstrap.cc:492] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: No bootstrap required, opened a new log
I20260927 21:36:16.690929 18699 ts_tablet_manager.cc:1397] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 21:36:16.691090 18700 log.cc:824] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:16.692728 18702 tablet_bootstrap.cc:654] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:16.693646 18702 log.cc:824] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:16.693885 18700 tablet_bootstrap.cc:492] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: No bootstrap required, opened a new log
I20260927 21:36:16.694108 18700 ts_tablet_manager.cc:1397] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:36:16.695830 18702 tablet_bootstrap.cc:492] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: No bootstrap required, opened a new log
I20260927 21:36:16.696084 18702 ts_tablet_manager.cc:1397] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 21:36:16.701323 18699 raft_consensus.cc:348] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.701772 18699 raft_consensus.cc:374] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:16.701861 18699 raft_consensus.cc:729] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94848b97740e407fb3bf19da014b1783, State: Initialized, Role: FOLLOWER
I20260927 21:36:16.702399 18699 consensus_queue.cc:260] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.702966 18702 raft_consensus.cc:348] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.703341 18702 raft_consensus.cc:374] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:16.703430 18702 raft_consensus.cc:729] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3601eb86fa4e18b6a135b62c0227de, State: Initialized, Role: FOLLOWER
I20260927 21:36:16.703552 18420 heartbeater.cc:503] Master 127.3.186.254:44503 was elected leader, sending a full tablet report...
I20260927 21:36:16.704105 18699 ts_tablet_manager.cc:1428] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 21:36:16.703982 18702 consensus_queue.cc:260] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.704023 18700 raft_consensus.cc:348] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.704404 18700 raft_consensus.cc:374] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:16.704501 18700 raft_consensus.cc:729] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e943a28179a94947b0dc79afe83dc490, State: Initialized, Role: FOLLOWER
I20260927 21:36:16.705147 18700 consensus_queue.cc:260] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.706648 18552 heartbeater.cc:503] Master 127.3.186.254:44503 was elected leader, sending a full tablet report...
I20260927 21:36:16.708400 18702 ts_tablet_manager.cc:1428] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 21:36:16.708405 18700 ts_tablet_manager.cc:1428] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260927 21:36:16.708958 18683 heartbeater.cc:503] Master 127.3.186.254:44503 was elected leader, sending a full tablet report...
W20260927 21:36:16.739055 18421 tablet.cc:2369] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:36:16.833314 18684 tablet.cc:2369] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:36:16.891047 18705 raft_consensus.cc:482] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:36:16.891256 18705 raft_consensus.cc:504] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.892705 18705 leader_election.cc:288] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e943a28179a94947b0dc79afe83dc490 (127.3.186.195:40879), 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269)
I20260927 21:36:16.898808 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.892982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:47831 (local address 127.3.186.195:40879)
0927 21:36:16.893188 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.893197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.893817 (+ 620us) server_negotiation.cc:407] Connection header received
0927 21:36:16.894027 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.894031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.894084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.894170 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:16.894704 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.895422 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.897120 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.897865 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.898061 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.898132 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.898443 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.898553 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:36:16.898941 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.893091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:43277 (local address 127.3.186.194:42269)
0927 21:36:16.893384 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.893391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.894060 (+ 669us) server_negotiation.cc:407] Connection header received
0927 21:36:16.894245 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.894248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.894301 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.894385 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:16.894955 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.895655 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.897267 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.898016 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.898180 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.898257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.898561 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.898690 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 21:36:16.899024 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.892954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:42269 (local address 127.3.186.193:43277)
0927 21:36:16.893963 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.893976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.894117 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.894397 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.894405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.894427 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:16.894808 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.894817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.895778 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.895784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.897126 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.897136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.898234 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.898241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.898411 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.898432 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.898524 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.898644 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":689,"threads_started":1}
I20260927 21:36:16.899359 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.892861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:40879 (local address 127.3.186.193:47831)
0927 21:36:16.893718 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:16.893732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:16.893893 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:16.894188 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:16.894192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:16.894216 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:16.894523 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.894535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.895584 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.895589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.896970 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:16.896980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.898074 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.898079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:16.898254 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.898283 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:36:16.898419 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:36:16.898531 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":1,"thread_start_us":463,"threads_started":1}
I20260927 21:36:16.899945 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d607de5ea34a479f0b3c6eb404839f" candidate_uuid: "94848b97740e407fb3bf19da014b1783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3601eb86fa4e18b6a135b62c0227de" is_pre_election: true
I20260927 21:36:16.900345 18507 raft_consensus.cc:2454] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94848b97740e407fb3bf19da014b1783 in term 0.
I20260927 21:36:16.900262 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d607de5ea34a479f0b3c6eb404839f" candidate_uuid: "94848b97740e407fb3bf19da014b1783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e943a28179a94947b0dc79afe83dc490" is_pre_election: true
I20260927 21:36:16.900681 18638 raft_consensus.cc:2454] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94848b97740e407fb3bf19da014b1783 in term 0.
I20260927 21:36:16.900915 18310 leader_election.cc:302] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3601eb86fa4e18b6a135b62c0227de, 94848b97740e407fb3bf19da014b1783; no voters:
W20260927 21:36:16.901257 18553 tablet.cc:2369] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:36:16.901297 18705 raft_consensus.cc:2781] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:36:16.901435 18705 raft_consensus.cc:482] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:36:16.901552 18705 raft_consensus.cc:3037] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:16.903834 18705 raft_consensus.cc:504] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.904405 18705 leader_election.cc:288] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [CANDIDATE]: Term 1 election: Requested vote from peers e943a28179a94947b0dc79afe83dc490 (127.3.186.195:40879), 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269)
I20260927 21:36:16.904742 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d607de5ea34a479f0b3c6eb404839f" candidate_uuid: "94848b97740e407fb3bf19da014b1783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e943a28179a94947b0dc79afe83dc490"
I20260927 21:36:16.904855 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d607de5ea34a479f0b3c6eb404839f" candidate_uuid: "94848b97740e407fb3bf19da014b1783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3601eb86fa4e18b6a135b62c0227de"
I20260927 21:36:16.904953 18638 raft_consensus.cc:3037] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:16.905045 18507 raft_consensus.cc:3037] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:16.908097 18638 raft_consensus.cc:2454] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94848b97740e407fb3bf19da014b1783 in term 1.
I20260927 21:36:16.908097 18507 raft_consensus.cc:2454] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94848b97740e407fb3bf19da014b1783 in term 1.
I20260927 21:36:16.908461 18310 leader_election.cc:302] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3601eb86fa4e18b6a135b62c0227de, 94848b97740e407fb3bf19da014b1783; no voters:
I20260927 21:36:16.908736 18705 raft_consensus.cc:2781] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:16.909161 18705 raft_consensus.cc:686] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 1 LEADER]: Becoming Leader. State: Replica: 94848b97740e407fb3bf19da014b1783, State: Running, Role: LEADER
I20260927 21:36:16.909726 18705 consensus_queue.cc:237] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } }
I20260927 21:36:16.914078 18234 catalog_manager.cc:5495] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94848b97740e407fb3bf19da014b1783 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "94848b97740e407fb3bf19da014b1783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94848b97740e407fb3bf19da014b1783" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 38205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 } health_report { overall_health: UNKNOWN } } }
I20260927 21:36:16.943610 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.939244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.3.186.193:38205)
0927 21:36:16.939391 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.939395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.939458 (+ 63us) server_negotiation.cc:407] Connection header received
0927 21:36:16.939607 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.939610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.939680 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.939790 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:36:16.940116 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.940705 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.941977 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.942354 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.942449 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.942796 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:36:16.943209 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:36:16.943305 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:16.943374 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:36:16.943425 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:36:16.950963 18507 raft_consensus.cc:1261] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Refusing update from remote peer 94848b97740e407fb3bf19da014b1783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:16.950963 18638 raft_consensus.cc:1261] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Refusing update from remote peer 94848b97740e407fb3bf19da014b1783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:16.952045 18710 consensus_queue.cc:1035] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3601eb86fa4e18b6a135b62c0227de" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:16.952378 18705 consensus_queue.cc:1035] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e943a28179a94947b0dc79afe83dc490" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:16.964359 18713 mvcc.cc:205] Tried to move back new op lower bound from 7334072225581617152 to 7334072225426702336. Current Snapshot: MvccSnapshot[applied={T|T < 7334072225581617152}]
I20260927 21:36:16.979660 18234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:16.979918 18234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:16.983888 18234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Sending DeleteTablet for 3 replicas of tablet 07d607de5ea34a479f0b3c6eb404839f
I20260927 21:36:16.985486 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d607de5ea34a479f0b3c6eb404839f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:16 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:36468
I20260927 21:36:16.985610 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18271
I20260927 21:36:16.986387 18723 tablet_replica.cc:330] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: stopping tablet replica
I20260927 21:36:16.987313 18723 raft_consensus.cc:2229] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:16.987296 18618 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d607de5ea34a479f0b3c6eb404839f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:16 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:45090
I20260927 21:36:16.987725 18723 raft_consensus.cc:2258] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:16.988116 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d607de5ea34a479f0b3c6eb404839f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:16 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:43728
I20260927 21:36:16.989058 18725 tablet_replica.cc:330] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: stopping tablet replica
I20260927 21:36:16.989404 18725 raft_consensus.cc:2229] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:36:16.989670 18723 ts_tablet_manager.cc:1905] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:16.989951 18725 raft_consensus.cc:2258] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:16.990427 18724 tablet_replica.cc:330] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: stopping tablet replica
I20260927 21:36:16.990754 18724 raft_consensus.cc:2229] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:16.991133 18724 raft_consensus.cc:2258] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:16.994333 18724 ts_tablet_manager.cc:1905] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:16.996300 18723 ts_tablet_manager.cc:1918] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:16.996444 18723 log.cc:1196] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/07d607de5ea34a479f0b3c6eb404839f
I20260927 21:36:16.997040 18723 ts_tablet_manager.cc:1939] T 07d607de5ea34a479f0b3c6eb404839f P 4d3601eb86fa4e18b6a135b62c0227de: Deleting consensus metadata
I20260927 21:36:16.998579 18220 catalog_manager.cc:4841] TS 4d3601eb86fa4e18b6a135b62c0227de (127.3.186.194:42269): tablet 07d607de5ea34a479f0b3c6eb404839f (table test-table [id=62e9a8483fa04af99c6cc5df3dd069ca]) successfully deleted
I20260927 21:36:16.998674 18725 ts_tablet_manager.cc:1905] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:17.002692 18724 ts_tablet_manager.cc:1918] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:17.002825 18724 log.cc:1196] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/07d607de5ea34a479f0b3c6eb404839f
I20260927 21:36:17.003362 18724 ts_tablet_manager.cc:1939] T 07d607de5ea34a479f0b3c6eb404839f P e943a28179a94947b0dc79afe83dc490: Deleting consensus metadata
I20260927 21:36:17.004808 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:16.986893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.3.186.193:38205)
0927 21:36:16.987043 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:16.987049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:16.987065 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:36:16.987110 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:16.987114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:16.988470 (+ 1356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:16.988570 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:16.990406 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.991510 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.993275 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:16.993960 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:16.994104 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:16.997371 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:16.997383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:16.997395 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:16.997434 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:17.001460 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:17.002256 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:17.002264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:17.002284 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:17.002671 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:17.003084 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:17.003089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:17.003092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:17.003586 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:17.003915 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:17.004225 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 21:36:17.004432 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:36:17.004851 18218 catalog_manager.cc:4841] TS e943a28179a94947b0dc79afe83dc490 (127.3.186.195:40879): tablet 07d607de5ea34a479f0b3c6eb404839f (table test-table [id=62e9a8483fa04af99c6cc5df3dd069ca]) successfully deleted
I20260927 21:36:17.005688 18415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:17.095534 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18271
W20260927 21:36:17.113366 18219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:38205 (error 108)
I20260927 21:36:17.113559 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18424
I20260927 21:36:17.115097 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.114694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43760)
0927 21:36:17.114852 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.114969 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:36:17.115319 18219 catalog_manager.cc:4584] TS 94848b97740e407fb3bf19da014b1783 (127.3.186.193:38205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d607de5ea34a479f0b3c6eb404839f: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
I20260927 21:36:17.131213 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.114451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.3.186.194:42269)
0927 21:36:17.114643 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:17.114649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:17.114667 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:36:17.114763 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:17.114768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:17.114849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:17.114958 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:17.116355 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:17.117063 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:17.118446 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:17.118838 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:17.118967 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:17.125848 (+ 6881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:17.125860 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:17.125874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:17.125909 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:17.128768 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:17.129319 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:17.129325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:17.129327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:17.129435 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:17.129912 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:17.129915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:17.129917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:17.130257 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:17.130467 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:17.130687 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 21:36:17.130865 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:36:17.131800 18547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:17.168587 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.168139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43774)
0927 21:36:17.168335 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.168439 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:36:17.169574 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.169159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43788)
0927 21:36:17.169303 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.169415 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:36:17.216609 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18424
I20260927 21:36:17.229931 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.229382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43796)
0927 21:36:17.229620 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.229725 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 21:36:17.230948 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.230524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43802)
0927 21:36:17.230693 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.230787 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:36:17.233942 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18555
I20260927 21:36:17.250593 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.234558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.3.186.195:40879)
0927 21:36:17.234723 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:17.234727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:17.234743 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:36:17.234851 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:17.234854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:17.234990 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:17.235097 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:17.236272 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:17.236904 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:17.238297 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:17.238664 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:17.238755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:17.245058 (+ 6303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:17.245071 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:17.245087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:17.245116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:17.247897 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:17.248558 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:17.248567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:17.248571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:17.248727 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:17.249116 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:17.249121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:17.249124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:17.249511 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:17.249770 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:17.250092 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 21:36:17.250283 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:36:17.251205 18678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:17.311131 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.310718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43806)
0927 21:36:17.310886 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.310975 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:36:17.312175 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:17.311784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:38205 (local address 127.0.0.1:43812)
0927 21:36:17.311982 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:17.312058 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:38205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 21:36:17.336241 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18555
I20260927 21:36:17.352257 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18201
I20260927 21:36:17.353268 18264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:17.460704 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18201
I20260927 21:36:17.481103 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44503
--webserver_interface=127.3.186.254
--webserver_port=40311
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:17.781404 18731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:17.781703 18731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:17.781872 18731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:17.791414 18731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:17.791529 18731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:17.791607 18731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:17.791706 18731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:17.791774 18731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:17.800483 18731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=40311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:17.801514 18731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:17.802667 18731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:17.814046 18736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:17.814040 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:17.814044 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:17.814953 18731 server_base.cc:1034] running on GCE node
I20260927 21:36:17.815630 18731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:17.816910 18731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:17.818141 18731 hybrid_clock.cc:648] HybridClock initialized: now 1790544977818084 us; error 44 us; skew 500 ppm
I20260927 21:36:17.818460 18731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:17 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544977, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:17.822016 18731 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:17.823814 18731 webserver.cc:460] Webserver started at http://127.3.186.254:40311/ using document root <none> and password file <none>
I20260927 21:36:17.824461 18731 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:17.824602 18731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:17.829500 18731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:36:17.832142 18746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:17.833281 18731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:36:17.833439 18731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "2b409676f4b74d888c62f4516747ece3"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "fcd24e7b09abe48e0a98e0d3b82dd2be"
server_key_iv: "0fcd104b7a2e37aa75c68b470a99414b"
server_key_version: "encryptionkey@0"
I20260927 21:36:17.834327 18731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:17.843719 18731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:17.847203 18731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:17.847447 18731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:17.864841 18731 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44503
I20260927 21:36:17.864923 18797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44503 every 8 connection(s)
I20260927 21:36:17.866140 18731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:17.869210 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18731
I20260927 21:36:17.870113 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:38205
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:44503
--webserver_port=43987
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:17.873631 18798 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:36:17.878386 18798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap starting.
I20260927 21:36:17.897209 18798 log.cc:824] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:17.912863 18798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:36:17.913347 18798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3: Bootstrap complete.
I20260927 21:36:17.922300 18798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:17.923311 18798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b409676f4b74d888c62f4516747ece3, State: Initialized, Role: FOLLOWER
I20260927 21:36:17.924080 18798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:17.924305 18798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:17.924443 18798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:17.924629 18798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:36:17.928524 18798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:17.929148 18798 leader_election.cc:302] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b409676f4b74d888c62f4516747ece3; no voters:
I20260927 21:36:17.929881 18798 leader_election.cc:288] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:36:17.930006 18802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:36:17.932180 18798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:17.933962 18802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [term 3 LEADER]: Becoming Leader. State: Replica: 2b409676f4b74d888c62f4516747ece3, State: Running, Role: LEADER
I20260927 21:36:17.934562 18802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } }
I20260927 21:36:17.937278 18803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b409676f4b74d888c62f4516747ece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } } }
I20260927 21:36:17.937691 18803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:17.937803 18804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b409676f4b74d888c62f4516747ece3. Latest consensus state: current_term: 3 leader_uuid: "2b409676f4b74d888c62f4516747ece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b409676f4b74d888c62f4516747ece3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44503 } } }
I20260927 21:36:17.938252 18804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b409676f4b74d888c62f4516747ece3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:17.942497 18808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:17.951016 18808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:17.952330 18808 catalog_manager.cc:1259] Loaded cluster ID: f90f3c0816e5425eb7eb41c08c680de4
I20260927 21:36:17.952423 18808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 21:36:17.954195 18808 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0ceee9d9ad google::LogMessage::Fail() at ??:0
@ 0x7f0ceeea088a google::LogMessage::SendToLog() at ??:0
@ 0x7f0ceee9d4de google::LogMessage::Flush() at ??:0
@ 0x7f0ceee9ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0d048d0512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0d048cd976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0d04955610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0d04955439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0cf1ae9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0cf1aec91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0cf1aec799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0cf1ac7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0cf4da16db start_thread at ??:0
@ 0x7f0cedd3761f clone at ??:0
W20260927 21:36:18.239425 18800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:18.239648 18800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:18.239728 18800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:18.246181 18800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:18.246259 18800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:18.246484 18800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:36:18.255165 18800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:38205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=43987
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:18.256206 18800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:18.257339 18800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:18.269490 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:18.269445 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:18.269914 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:18.270313 18800 server_base.cc:1034] running on GCE node
I20260927 21:36:18.270879 18800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:18.271708 18800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:18.272925 18800 hybrid_clock.cc:648] HybridClock initialized: now 1790544978272885 us; error 33 us; skew 500 ppm
I20260927 21:36:18.273245 18800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:18 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544978, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:18.276194 18800 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:36:18.277413 18800 webserver.cc:460] Webserver started at http://127.3.186.193:43987/ using document root <none> and password file <none>
I20260927 21:36:18.278035 18800 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:18.278129 18800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:18.282644 18800 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:36:18.285193 18846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:18.286458 18800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:36:18.286618 18800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "94848b97740e407fb3bf19da014b1783"
format_stamp: "Formatted at 2026-09-27 21:36:12 on dist-test-slave-pqr8"
server_key: "06a4a59bc6f2b0a84131ece231fea92b"
server_key_iv: "6d76a22b831125b03c11ee92c33899e6"
server_key_version: "encryptionkey@0"
I20260927 21:36:18.287513 18800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:18.315646 18800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:18.319098 18800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:18.319360 18800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:18.320190 18800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:18.321848 18853 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 21:36:18.326668 18853 ts_tablet_manager.cc:1852] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 21:36:18.327581 18853 ts_tablet_manager.cc:1905] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:18.334167 18853 ts_tablet_manager.cc:1918] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:18.334353 18853 log.cc:1196] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/07d607de5ea34a479f0b3c6eb404839f
I20260927 21:36:18.334786 18853 ts_tablet_manager.cc:1939] T 07d607de5ea34a479f0b3c6eb404839f P 94848b97740e407fb3bf19da014b1783: Deleting consensus metadata
I20260927 21:36:18.335151 18800 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 21:36:18.335265 18800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260927 21:36:18.335428 18800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:18.335513 18800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:18.367816 18800 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:38205
I20260927 21:36:18.367900 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:38205 every 8 connection(s)
I20260927 21:36:18.369089 18800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:36:18.373262 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:18.371723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:51251)
0927 21:36:18.372336 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:18.372553 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":236,"threads_started":1}
I20260927 21:36:18.373888 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18800
W20260927 21:36:18.373970 18960 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44503 (0 consecutive failures): Network error: Failed to ping master at 127.3.186.254:44503: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
I20260927 21:36:18.374540 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:42269
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:44503
--webserver_port=32945
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:18.374758 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:18.374385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:38089)
0927 21:36:18.374528 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:18.374606 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:36:18.375573 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:18.375270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:42511)
0927 21:36:18.375395 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:18.375464 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:36:18.375860 18960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:36:18.671363 18963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:18.671577 18963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:18.671713 18963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:18.678072 18963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:18.678149 18963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:18.678300 18963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:36:18.686924 18963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:42269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=32945
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:18.687923 18963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:18.689039 18963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:18.700954 18968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:18.701016 18971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:18.701035 18969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:18.701566 18963 server_base.cc:1034] running on GCE node
I20260927 21:36:18.702052 18963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:18.702836 18963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:18.704027 18963 hybrid_clock.cc:648] HybridClock initialized: now 1790544978703976 us; error 47 us; skew 500 ppm
I20260927 21:36:18.704303 18963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:18 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544978, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:18.707396 18963 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:36:18.708585 18963 webserver.cc:460] Webserver started at http://127.3.186.194:32945/ using document root <none> and password file <none>
I20260927 21:36:18.709200 18963 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:18.709337 18963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:18.714309 18963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:36:18.716665 18978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:18.717851 18963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:36:18.718025 18963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "4d3601eb86fa4e18b6a135b62c0227de"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "f4393f1e1d3728f874108f0917d70c5a"
server_key_iv: "887770ff644218b3f3469d837eabc1fc"
server_key_version: "encryptionkey@0"
I20260927 21:36:18.718891 18963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:18.736316 18963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:18.739998 18963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:18.740231 18963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:18.741070 18963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:18.742163 18963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:18.742270 18963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:18.742386 18963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:18.742465 18963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:18.773972 18963 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:42269
I20260927 21:36:18.774039 19090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:42269 every 8 connection(s)
I20260927 21:36:18.775264 18963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:36:18.779577 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:18.777964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:55513)
0927 21:36:18.778608 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:18.778799 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
W20260927 21:36:18.780192 19091 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44503 (0 consecutive failures): Network error: Failed to ping master at 127.3.186.254:44503: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
I20260927 21:36:18.780970 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:18.780656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:55999)
0927 21:36:18.780797 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:18.780862 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:36:18.781733 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:18.781417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:56487)
0927 21:36:18.781564 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:18.781638 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 21:36:18.781991 19091 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:36:18.783573 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 18963
I20260927 21:36:18.784050 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:40879
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:44503
--webserver_port=34073
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:19.082988 19094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:19.083266 19094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:19.083417 19094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:19.090165 19094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:19.090291 19094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:19.090446 19094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:36:19.099313 19094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:40879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=34073
--tserver_master_addrs=127.3.186.254:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:19.100386 19094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:19.101598 19094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:19.113791 19100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:19.114017 19099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:19.113788 19102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:19.114706 19094 server_base.cc:1034] running on GCE node
I20260927 21:36:19.115276 19094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:19.116110 19094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:19.117340 19094 hybrid_clock.cc:648] HybridClock initialized: now 1790544979117287 us; error 46 us; skew 500 ppm
I20260927 21:36:19.117697 19094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:19 dist-test-slave-pqr8 krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544979, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:19.120772 19094 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:36:19.122258 19094 webserver.cc:460] Webserver started at http://127.3.186.195:34073/ using document root <none> and password file <none>
I20260927 21:36:19.122903 19094 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:19.122996 19094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:19.127681 19094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:36:19.130019 19109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:19.131170 19094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:36:19.131330 19094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "e943a28179a94947b0dc79afe83dc490"
format_stamp: "Formatted at 2026-09-27 21:36:13 on dist-test-slave-pqr8"
server_key: "18250524d50f7ae28c0d6c45a049aac0"
server_key_iv: "9afe50cbd262e599b4ae476af0d02ef5"
server_key_version: "encryptionkey@0"
I20260927 21:36:19.132205 19094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:19.142272 19094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:19.146103 19094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:19.146345 19094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:19.147187 19094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:19.148293 19094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:19.148422 19094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:19.148547 19094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:19.148631 19094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:19.179863 19094 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:40879
I20260927 21:36:19.179943 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:40879 every 8 connection(s)
I20260927 21:36:19.181149 19094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:36:19.183050 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19094
I20260927 21:36:19.185817 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:19.183840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:39167)
0927 21:36:19.184689 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:19.184980 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":343,"threads_started":1}
W20260927 21:36:19.186486 19222 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:44503 (0 consecutive failures): Network error: Failed to ping master at 127.3.186.254:44503: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
I20260927 21:36:19.187191 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:19.186920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:46461)
0927 21:36:19.187041 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:19.187100 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:36:19.187945 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:19.187679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:50521)
0927 21:36:19.187783 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:19.187848 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 21:36:19.188207 19222 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:36:19.377041 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:19.376318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:39253)
0927 21:36:19.376736 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:19.376847 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260927 21:36:19.783655 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:19.782574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:47403)
0927 21:36:19.783180 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:19.783320 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":273,"threads_started":1}
I20260927 21:36:20.189384 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:20.188656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:53631)
0927 21:36:20.189102 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:20.189184 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260927 21:36:20.378494 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:20.377866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:46899)
0927 21:36:20.378293 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:20.378365 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260927 21:36:20.785058 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:20.784441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:39413)
0927 21:36:20.784843 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:20.784912 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":203,"threads_started":1}
I20260927 21:36:21.190888 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:21.190178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:46643)
0927 21:36:21.190616 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:21.190687 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260927 21:36:21.379937 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:21.379306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:41177)
0927 21:36:21.379695 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:21.379768 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":204,"threads_started":1}
I20260927 21:36:21.786832 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:21.785911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:33939)
0927 21:36:21.786483 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:21.786587 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":308,"threads_started":1}
I20260927 21:36:22.192246 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:22.191647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:53545)
0927 21:36:22.192011 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:22.192097 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":197,"threads_started":1}
I20260927 21:36:22.381392 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:22.380678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:32913)
0927 21:36:22.381167 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:22.381236 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":302,"threads_started":1}
I20260927 21:36:22.788328 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:22.787597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:38907)
0927 21:36:22.788077 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:22.788158 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
I20260927 21:36:23.193666 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:23.192992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:34169)
0927 21:36:23.193373 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:23.193440 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":200,"threads_started":1}
I20260927 21:36:23.382761 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:23.382160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:33795)
0927 21:36:23.382549 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:23.382617 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":198,"threads_started":1}
I20260927 21:36:23.789855 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:23.789157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:39625)
0927 21:36:23.789611 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:23.789684 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":196,"threads_started":1}
I20260927 21:36:24.195152 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:24.194523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:47339)
0927 21:36:24.194931 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:24.195004 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":197,"threads_started":1}
I20260927 21:36:24.384114 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:24.383513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:56639)
0927 21:36:24.383901 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:24.383975 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":196,"threads_started":1}
I20260927 21:36:24.791366 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:24.790664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:42055)
0927 21:36:24.791086 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:24.791169 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260927 21:36:25.196554 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:25.195947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:55121)
0927 21:36:25.196336 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:25.196414 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":203,"threads_started":1}
I20260927 21:36:25.385743 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:25.384920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:42199)
0927 21:36:25.385375 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:25.385448 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":214,"threads_started":1}
I20260927 21:36:25.794235 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:25.792282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:58651)
0927 21:36:25.793898 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:25.793986 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":1322,"threads_started":1}
I20260927 21:36:26.198112 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:26.197330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:58121)
0927 21:36:26.197851 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:26.197920 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":274,"threads_started":1}
I20260927 21:36:26.387276 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:26.386566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:49805)
0927 21:36:26.387041 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:26.387115 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":297,"threads_started":1}
I20260927 21:36:26.795609 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:26.794970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:42815)
0927 21:36:26.795380 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:26.795458 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
I20260927 21:36:27.199484 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:27.198857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:49625)
0927 21:36:27.199261 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:27.199338 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260927 21:36:27.388825 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:27.388187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:42983)
0927 21:36:27.388581 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:27.388672 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":194,"threads_started":1}
I20260927 21:36:27.796977 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:27.796355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:40255)
0927 21:36:27.796760 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:27.796829 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":193,"threads_started":1}
I20260927 21:36:28.200915 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:28.200289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:51459)
0927 21:36:28.200700 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:28.200769 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260927 21:36:28.390288 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:28.389701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:36943)
0927 21:36:28.390072 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:28.390140 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":191,"threads_started":1}
I20260927 21:36:28.798473 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:28.797883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:60321)
0927 21:36:28.798264 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:28.798334 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":214,"threads_started":1}
I20260927 21:36:29.202365 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:29.201772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:56515)
0927 21:36:29.202145 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:29.202224 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":198,"threads_started":1}
I20260927 21:36:29.391671 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:29.391055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:39189)
0927 21:36:29.391437 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:29.391511 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":204,"threads_started":1}
I20260927 21:36:29.799988 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:29.799331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:36449)
0927 21:36:29.799738 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:29.799821 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":211,"threads_started":1}
I20260927 21:36:30.203831 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:30.203146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:51557)
0927 21:36:30.203562 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:30.203640 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260927 21:36:30.393002 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:30.392396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:39833)
0927 21:36:30.392784 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:30.392857 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":216,"threads_started":1}
I20260927 21:36:30.801577 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:30.800886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:49853)
0927 21:36:30.801306 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:30.801380 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260927 21:36:31.205303 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:31.204676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:36029)
0927 21:36:31.205087 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:31.205169 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260927 21:36:31.394490 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:31.393822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:52299)
0927 21:36:31.394228 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:31.394320 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":197,"threads_started":1}
I20260927 21:36:31.803279 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:31.802399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:55869)
0927 21:36:31.802943 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:31.803055 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":269,"threads_started":1}
I20260927 21:36:32.206856 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:32.206089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:57365)
0927 21:36:32.206555 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:32.206706 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":239,"threads_started":1}
I20260927 21:36:32.395953 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:32.395246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:49859)
0927 21:36:32.395660 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:32.395787 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":193,"threads_started":1}
I20260927 21:36:32.804663 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:32.803998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:38705)
0927 21:36:32.804431 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:32.804505 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":208,"threads_started":1}
I20260927 21:36:33.208173 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:33.207571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:39059)
0927 21:36:33.207958 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:33.208031 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":206,"threads_started":1}
I20260927 21:36:33.397720 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:33.397012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.193:45131)
0927 21:36:33.397417 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:33.397498 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260927 21:36:33.806522 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:33.805703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.194:52025)
0927 21:36:33.806203 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:33.806305 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":239,"threads_started":1}
I20260927 21:36:34.184470 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18800
I20260927 21:36:34.207029 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:34.185442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.3.186.193:38205)
0927 21:36:34.186100 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:34.186124 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:34.186174 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:36:34.187863 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:34.187872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:34.188282 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:34.188548 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:34.189782 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:34.191038 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:34.192313 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:34.192914 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:34.193035 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:34.201332 (+ 8297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:34.201340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:34.201352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:34.201386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:34.204243 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:34.204827 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:34.204833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:34.204838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:34.204990 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:34.205331 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:34.205334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:34.205336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:34.205871 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:34.206089 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:34.206390 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:36:34.206769 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":362,"threads_started":1}
I20260927 21:36:34.208245 18955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:34.210217 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:34.209360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44503 (local address 127.3.186.195:60505)
0927 21:36:34.209896 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:34.210014 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":311,"threads_started":1}
I20260927 21:36:34.289076 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18800
I20260927 21:36:34.305702 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 18963
I20260927 21:36:34.322508 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:34.306687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.3.186.194:42269)
0927 21:36:34.307316 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:34.307332 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:34.307374 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:36:34.308791 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:34.308799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:34.309170 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:34.309502 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:34.310867 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:34.311967 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:34.313169 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:34.313869 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:34.314056 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:34.316490 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:34.316498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:34.316510 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:34.316541 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:34.319276 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:34.319951 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:34.319966 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:34.319972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:34.320128 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:34.320505 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:34.320511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:34.320514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:34.320972 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:34.321282 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:34.321710 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 21:36:34.322088 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":327,"threads_started":1}
I20260927 21:36:34.323714 19086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:34.401798 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 18963
I20260927 21:36:34.418092 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19094
I20260927 21:36:34.434650 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:34.419105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.3.186.195:40879)
0927 21:36:34.419728 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:34.419742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:34.419779 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:36:34.421306 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:34.421314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:34.421808 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:34.422082 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:34.423442 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:34.424575 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:34.425897 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:34.426508 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:34.426667 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:34.429051 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:34.429057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:34.429070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:34.429112 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:34.431963 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:34.432551 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:34.432557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:34.432562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:34.432704 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:34.433009 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:34.433012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:34.433014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:34.433452 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:34.433734 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:34.434011 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:36:34.434375 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260927 21:36:34.435904 19217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:34.512688 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19094
2026-09-27T21:36:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23900 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:36:34 dist-test-slave-pqr8 krb5kdc[19279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:36:34 dist-test-slave-pqr8 krb5kdc[19279](info): set up 2 sockets
Sep 27 21:36:34 dist-test-slave-pqr8 krb5kdc[19279](info): commencing operation
krb5kdc: starting...
W20260927 21:36:35.593986 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:36:35 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:36:35.798359 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46229
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:36.103377 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:36.103638 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:36.103765 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:36.110625 19292 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:36:36.110702 19292 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:36:36.110772 19292 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:36:36.110869 19292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:36.110951 19292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:36.111061 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:36.111121 19292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:36.111209 19292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:36.120360 19292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:36.121436 19292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:36.122689 19292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:36.134186 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:36.134265 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:36.134238 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:36.134900 19292 server_base.cc:1034] running on GCE node
I20260927 21:36:36.135612 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:36.136830 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:36.138056 19292 hybrid_clock.cc:648] HybridClock initialized: now 1790544996137993 us; error 42 us; skew 500 ppm
I20260927 21:36:36.138361 19292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:36 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544996, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:36.141252 19292 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:36.142580 19292 webserver.cc:460] Webserver started at http://127.3.186.254:38647/ using document root <none> and password file <none>
I20260927 21:36:36.143153 19292 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:36.143234 19292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:36.143456 19292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:36.146142 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "888c4474e23544ebbf48c164e5a18991"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "53a4be3df34194846eaf3dc1c644708e"
server_key_iv: "b5f4eb58e7170f482e09977ddb3efc76"
server_key_version: "encryptionkey@0"
I20260927 21:36:36.146843 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "888c4474e23544ebbf48c164e5a18991"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "53a4be3df34194846eaf3dc1c644708e"
server_key_iv: "b5f4eb58e7170f482e09977ddb3efc76"
server_key_version: "encryptionkey@0"
I20260927 21:36:36.151397 19292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:36:36.154466 19307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:36.155656 19292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:36:36.155841 19292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "888c4474e23544ebbf48c164e5a18991"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "53a4be3df34194846eaf3dc1c644708e"
server_key_iv: "b5f4eb58e7170f482e09977ddb3efc76"
server_key_version: "encryptionkey@0"
I20260927 21:36:36.156030 19292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:36.174028 19292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:36.177837 19292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:36.178105 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:36.194954 19292 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:46229
I20260927 21:36:36.194983 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:46229 every 8 connection(s)
I20260927 21:36:36.196125 19292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:36.199411 19359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:36.205294 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19292
I20260927 21:36:36.205521 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:36:36.206362 3819 external_mini_cluster.cc:1428] Setting key 798e9417d96bbeae448517ebec6e5aa4
I20260927 21:36:36.209685 19359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap starting.
I20260927 21:36:36.212826 19359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:36.213841 19359 log.cc:824] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:36.216255 19359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: No bootstrap required, opened a new log
Sep 27 21:36:36 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:36.222958 19359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:36.223289 19359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:36.223392 19359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888c4474e23544ebbf48c164e5a18991, State: Initialized, Role: FOLLOWER
I20260927 21:36:36.223942 19359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:36.224175 19359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:36.224316 19359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:36.224509 19359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:36.226941 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:36.208179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.3.186.254:46229)
0927 21:36:36.208871 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:36.208881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:36.208916 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:36:36.210499 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:36.210517 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:36.210850 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:36.211132 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:36.212743 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:36.214006 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:36.215603 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:36.216210 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:36.216417 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:36.220093 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:36.220103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:36.220115 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:36.220149 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:36.222958 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:36.223736 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:36.223743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:36.223748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:36.223905 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:36.224297 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:36.224307 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:36.224312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:36.224857 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:36.225202 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:36.225642 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 21:36:36.226072 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":336,"threads_started":1}
I20260927 21:36:36.226961 19359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:36.227466 19359 leader_election.cc:302] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888c4474e23544ebbf48c164e5a18991; no voters:
I20260927 21:36:36.228099 19359 leader_election.cc:288] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:36:36.228240 19364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:36.229406 19364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 LEADER]: Becoming Leader. State: Replica: 888c4474e23544ebbf48c164e5a18991, State: Running, Role: LEADER
I20260927 21:36:36.230190 19359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:36.230271 19364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:36.234555 19366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888c4474e23544ebbf48c164e5a18991. Latest consensus state: current_term: 1 leader_uuid: "888c4474e23544ebbf48c164e5a18991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } } }
I20260927 21:36:36.235111 19366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:36.234521 19365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "888c4474e23544ebbf48c164e5a18991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } } }
I20260927 21:36:36.235596 19365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:36.239414 19375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:36.245386 19375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:36.253587 19375 catalog_manager.cc:1347] Generated new cluster ID: 3876921eec67431e8cf75afefea7ba53
I20260927 21:36:36.253684 19375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:36.262033 19375 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:36:36.263129 19375 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:36.269737 19375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Generated new TSK 0
I20260927 21:36:36.270351 19375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:36:36.333331 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46229
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:36.651648 19386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:36.651882 19386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:36.651968 19386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:36.658569 19386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:36.658656 19386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:36.658762 19386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:36:36.667893 19386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:36.668869 19386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:36.669978 19386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:36.681705 19391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:36.681751 19392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:36.681859 19394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:36.682797 19386 server_base.cc:1034] running on GCE node
I20260927 21:36:36.683359 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:36.684118 19386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:36.685307 19386 hybrid_clock.cc:648] HybridClock initialized: now 1790544996685253 us; error 53 us; skew 500 ppm
I20260927 21:36:36.685601 19386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:36 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544996, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:36.688557 19386 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:36:36.689801 19386 webserver.cc:460] Webserver started at http://127.3.186.193:36437/ using document root <none> and password file <none>
I20260927 21:36:36.690344 19386 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:36.690436 19386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:36.690680 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:36.693243 19386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "baf994a8535b470b85f0fe962e5cfda6"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "9173fecedf4468bd3e3bce161e56d945"
server_key_iv: "391201c258f3625645f408f64fe7cf22"
server_key_version: "encryptionkey@0"
I20260927 21:36:36.693955 19386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "baf994a8535b470b85f0fe962e5cfda6"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "9173fecedf4468bd3e3bce161e56d945"
server_key_iv: "391201c258f3625645f408f64fe7cf22"
server_key_version: "encryptionkey@0"
I20260927 21:36:36.698534 19386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:36:36.701658 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:36.702797 19386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:36:36.702960 19386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "baf994a8535b470b85f0fe962e5cfda6"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "9173fecedf4468bd3e3bce161e56d945"
server_key_iv: "391201c258f3625645f408f64fe7cf22"
server_key_version: "encryptionkey@0"
I20260927 21:36:36.703150 19386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:36.724316 19386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:36.728434 19386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:36.728711 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:36.729664 19386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:36.730885 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:36.730984 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:36.731117 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:36.731191 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:36.762584 19386 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:33895
I20260927 21:36:36.762655 19513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:33895 every 8 connection(s)
I20260927 21:36:36.763639 19386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:36:36.772523 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19386
I20260927 21:36:36.772720 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:36:36.773564 3819 external_mini_cluster.cc:1428] Setting key bb59d4e4f56e42971411e43c347cf36f
Sep 27 21:36:36 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544996, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:36.783334 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:36.765975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:53157 (local address 127.3.186.254:46229)
0927 21:36:36.766395 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:36.766399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:36.767302 (+ 903us) server_negotiation.cc:407] Connection header received
0927 21:36:36.768969 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:36.768978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:36.769083 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:36.769193 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:36.770673 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:36.771368 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:36.772892 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:36.773560 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:36.773712 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:36.777112 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:36.777121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:36.777128 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:36.777169 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:36.780374 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:36.781019 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:36.781027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:36.781033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:36.781171 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:36.781561 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:36.781568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:36.781573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:36.781936 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:36.782082 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:36.782695 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 21:36:36.782915 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":212,"threads_started":1}
I20260927 21:36:36.783789 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:36.766518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:53157)
0927 21:36:36.767153 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:36.767186 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:36.768727 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:36.769396 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:36.769403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:36.770046 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:36.770525 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:36.770540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:36.771532 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:36.771535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:36.772738 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:36.772750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:36.773797 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:36.773801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:36.773971 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:36.774622 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:36.774650 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:36.776899 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:36.780506 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:36.780511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:36.780521 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:36.780899 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:36.781277 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:36.781280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:36.781282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:36.781436 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:36.782083 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:36.782088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:36.782342 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 21:36:36.782545 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 21:36:36.782858 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":257,"threads_started":1}
I20260927 21:36:36.785261 19514 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46229
I20260927 21:36:36.785583 19514 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:36.786224 19514 heartbeater.cc:511] Master 127.3.186.254:46229 requested a full tablet report, sending...
I20260927 21:36:36.788106 19324 ts_manager.cc:194] Registered new tserver with Master: baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193:33895)
I20260927 21:36:36.789429 19324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:53157
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:36:36.822458 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46229
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:37.119505 19520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:37.119724 19520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:37.119874 19520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:37.126653 19520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:37.126766 19520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:37.126943 19520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:36:37.135805 19520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:37.136832 19520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:37.138051 19520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:37.150436 19525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:37.150403 19526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:37.150995 19528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:37.151576 19520 server_base.cc:1034] running on GCE node
I20260927 21:36:37.152261 19520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:37.153323 19520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:37.154536 19520 hybrid_clock.cc:648] HybridClock initialized: now 1790544997154439 us; error 107 us; skew 500 ppm
I20260927 21:36:37.154778 19520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:37 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:37.157634 19520 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:36:37.158914 19520 webserver.cc:460] Webserver started at http://127.3.186.194:40333/ using document root <none> and password file <none>
I20260927 21:36:37.159571 19520 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:37.159693 19520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:37.159976 19520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:37.162650 19520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "59679a36864e494fa4a99de51e0b302a"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "5d325ad9c35a87ed7cd7666416c3e103"
server_key_iv: "e9e5e7add2b7626971fd2dda67a1c7cc"
server_key_version: "encryptionkey@0"
I20260927 21:36:37.163353 19520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "59679a36864e494fa4a99de51e0b302a"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "5d325ad9c35a87ed7cd7666416c3e103"
server_key_iv: "e9e5e7add2b7626971fd2dda67a1c7cc"
server_key_version: "encryptionkey@0"
I20260927 21:36:37.167902 19520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:36:37.170922 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:37.172048 19520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:36:37.172214 19520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "59679a36864e494fa4a99de51e0b302a"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "5d325ad9c35a87ed7cd7666416c3e103"
server_key_iv: "e9e5e7add2b7626971fd2dda67a1c7cc"
server_key_version: "encryptionkey@0"
I20260927 21:36:37.172380 19520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:37.187922 19520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:37.191344 19520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:37.191593 19520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:37.192386 19520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:37.193565 19520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:37.193645 19520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:37.193802 19520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:37.193850 19520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:37.229743 19520 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:46117
I20260927 21:36:37.229815 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:46117 every 8 connection(s)
I20260927 21:36:37.230954 19520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:36:37.239728 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19520
I20260927 21:36:37.239908 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:36:37.240373 3819 external_mini_cluster.cc:1428] Setting key 771870f3e970adc756fd4c4e3ce9cb29
Sep 27 21:36:37 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:37.249657 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.233233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:59085 (local address 127.3.186.254:46229)
0927 21:36:37.233384 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:37.233388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:37.234378 (+ 990us) server_negotiation.cc:407] Connection header received
0927 21:36:37.236023 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:37.236029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:37.236127 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:37.236237 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:37.237599 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.238311 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.240072 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.240533 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.240632 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.244173 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:37.244184 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:37.244188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:37.244215 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:37.246806 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:37.247466 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:37.247472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:37.247475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:37.247597 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:37.248056 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:37.248060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:37.248063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:37.248309 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:37.248456 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:37.249071 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 21:36:37.249277 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:36:37.250429 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.233657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:59085)
0927 21:36:37.234240 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:37.234280 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:37.235780 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:37.236465 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:37.236472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:37.236910 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:37.237421 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.237436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.238473 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.238477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.239945 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.239960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.240743 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.240752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.241012 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.241666 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:37.241690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:37.243969 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:37.246942 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:37.246951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:37.246964 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:37.247345 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:37.247721 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:37.247726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:37.247729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:37.247959 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:37.248431 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:37.248438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:37.248726 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 21:36:37.248961 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 21:36:37.249413 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":231,"threads_started":1}
I20260927 21:36:37.251794 19648 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46229
I20260927 21:36:37.252054 19648 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:37.252707 19648 heartbeater.cc:511] Master 127.3.186.254:46229 requested a full tablet report, sending...
I20260927 21:36:37.253991 19324 ts_manager.cc:194] Registered new tserver with Master: 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117)
I20260927 21:36:37.254730 19324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:59085
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:36:37.288203 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46229
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:37.589427 19653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:37.589670 19653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:37.589823 19653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:37.596411 19653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:37.596496 19653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:37.596776 19653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:36:37.605608 19653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:37.606740 19653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:37.607952 19653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:37.620059 19658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:37.620095 19661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:37.620024 19659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:37.620797 19653 server_base.cc:1034] running on GCE node
I20260927 21:36:37.621372 19653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:37.622195 19653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:37.623404 19653 hybrid_clock.cc:648] HybridClock initialized: now 1790544997623369 us; error 49 us; skew 500 ppm
I20260927 21:36:37.623683 19653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:37 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:37.626639 19653 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:36:37.627849 19653 webserver.cc:460] Webserver started at http://127.3.186.195:40837/ using document root <none> and password file <none>
I20260927 21:36:37.628399 19653 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:37.628492 19653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:37.628721 19653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:36:37.631335 19653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "687e89e535894539865ea65f0a276287"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "8190892c134bd5e852afc21c03e4a45d"
server_key_iv: "e47296414f83403f8627e6ef86cda7cd"
server_key_version: "encryptionkey@0"
I20260927 21:36:37.632023 19653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "687e89e535894539865ea65f0a276287"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "8190892c134bd5e852afc21c03e4a45d"
server_key_iv: "e47296414f83403f8627e6ef86cda7cd"
server_key_version: "encryptionkey@0"
I20260927 21:36:37.636588 19653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 21:36:37.639675 19668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:37.640908 19653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:36:37.641126 19653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "687e89e535894539865ea65f0a276287"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "8190892c134bd5e852afc21c03e4a45d"
server_key_iv: "e47296414f83403f8627e6ef86cda7cd"
server_key_version: "encryptionkey@0"
I20260927 21:36:37.641320 19653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:37.668726 19653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:37.672022 19653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:37.672294 19653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:37.673116 19653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:37.674273 19653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:37.674340 19653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:37.674481 19653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:37.674533 19653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:37.705572 19653 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:44717
I20260927 21:36:37.705701 19780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:44717 every 8 connection(s)
I20260927 21:36:37.706878 19653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:36:37.716066 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19653
I20260927 21:36:37.716246 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:36:37.716820 3819 external_mini_cluster.cc:1428] Setting key abbaa3063961ffc27885e83629ce8e77
Sep 27 21:36:37 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:37.725922 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.709172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:37167 (local address 127.3.186.254:46229)
0927 21:36:37.709373 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:37.709379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:37.710450 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 21:36:37.712041 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:37.712045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:37.712115 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:37.712198 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:37.713439 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.714222 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.715629 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.716271 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.716423 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.720405 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:37.720410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:37.720413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:37.720442 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:37.722996 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:37.723727 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:37.723732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:37.723736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:37.723858 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:37.724268 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:37.724274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:37.724277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:37.724559 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:37.724708 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:37.725333 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 21:36:37.725592 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:36:37.726712 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.709685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:37167)
0927 21:36:37.710301 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:37.710339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:37.711814 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:37.712376 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:37.712383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:37.712811 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:37.713303 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.713319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.714354 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.714358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.715512 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.715523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.716493 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.716497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.716769 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.717752 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:37.717778 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:37.720217 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:37.723171 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:37.723179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:37.723193 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:37.723601 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:37.723965 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:37.723970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:37.723974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:37.724155 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:37.724684 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:37.724690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:37.724971 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 21:36:37.725237 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 21:36:37.725702 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":256,"threads_started":1}
I20260927 21:36:37.727847 19781 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46229
I20260927 21:36:37.728119 19781 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:37.728755 19781 heartbeater.cc:511] Master 127.3.186.254:46229 requested a full tablet report, sending...
I20260927 21:36:37.730180 19324 ts_manager.cc:194] Registered new tserver with Master: 687e89e535894539865ea65f0a276287 (127.3.186.195:44717)
I20260927 21:36:37.731184 19324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:37167
I20260927 21:36:37.732262 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:36:37.748121 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.736454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.3.186.254:46229)
0927 21:36:37.736594 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:37.736598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:37.737102 (+ 504us) server_negotiation.cc:407] Connection header received
0927 21:36:37.737219 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:37.737223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:37.737273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:37.737348 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:37.738670 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.739317 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.740685 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.741040 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.741192 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.743434 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:37.743439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:37.743442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:37.743471 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:37.746001 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:37.746636 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:37.746640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:37.746655 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:37.746764 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:37.747057 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:37.747060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:37.747062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:37.747308 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:37.747472 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:37.747781 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 21:36:37.747923 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:36:37.751524 19324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:36:37.758097 19324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:36:37.793295 19514 heartbeater.cc:503] Master 127.3.186.254:46229 was elected leader, sending a full tablet report...
I20260927 21:36:37.797220 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.769401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.3.186.194:46117)
0927 21:36:37.785168 (+ 15767us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:37.785180 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:37.786183 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 21:36:37.787222 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:37.787230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:37.787438 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:37.788435 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:37.789208 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.790684 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.794023 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.795679 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.795802 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.796011 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:37.796742 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 21:36:37.796921 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15524,"thread_start_us":15781,"threads_started":1}
I20260927 21:36:37.798900 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.769234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:46117 (local address 127.0.0.1:54574)
0927 21:36:37.785989 (+ 16755us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:37.786045 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:37.787041 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:37.788702 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:37.788708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:37.788735 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:37.789040 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.789049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.790850 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.790855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.793836 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.793852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.796233 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.796241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.796474 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.796530 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:36:37.797629 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0927 21:36:37.797802 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16454,"thread_start_us":16311,"threads_started":1}
I20260927 21:36:37.802790 19583 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8109b99760412fbdc911eb18378c1a (DEFAULT_TABLE table=test-table [id=d475b98da63342369d59d9ce58115548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:37.804224 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.786568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.3.186.193:33895)
0927 21:36:37.795549 (+ 8981us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:37.795558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:37.796065 (+ 507us) server_negotiation.cc:407] Connection header received
0927 21:36:37.796283 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:37.796291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:37.796494 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:37.796683 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:37.797425 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.799116 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.801502 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.802926 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.803021 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.803213 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:37.803765 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 21:36:37.803924 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8797,"thread_start_us":8674,"threads_started":1}
I20260927 21:36:37.804230 19583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8109b99760412fbdc911eb18378c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:37.804864 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.792401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33895 (local address 127.0.0.1:37464)
0927 21:36:37.795954 (+ 3553us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:37.795967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:37.796111 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:37.796873 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:37.796877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:37.796900 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:37.797223 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.797235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.799294 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.799299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.801343 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.801357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.803398 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.803403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.803546 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.803587 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:36:37.804445 (+ 858us) client_negotiation.cc:240] Negotiation successful
0927 21:36:37.804577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3440,"thread_start_us":3327,"threads_started":1}
I20260927 21:36:37.806119 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.769713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:44717 (local address 127.0.0.1:37916)
0927 21:36:37.786709 (+ 16996us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:37.786724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:37.786866 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:37.799443 (+ 12577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:37.799448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:37.799481 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:37.799813 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.799829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.801820 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.801825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.803732 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:37.803743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.805406 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.805411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:37.805585 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.805616 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:36:37.805676 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:36:37.805790 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16891,"thread_start_us":16634,"threads_started":1}
I20260927 21:36:37.807394 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:37.770315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.3.186.195:44717)
0927 21:36:37.798763 (+ 28448us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:37.798770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:37.798798 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:36:37.798878 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:37.798884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:37.799087 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:37.800003 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:37.800253 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.801679 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.803912 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:37.805208 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:37.806572 (+ 1364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:37.806828 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:37.806901 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 21:36:37.807042 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28292,"thread_start_us":28882,"threads_started":1}
I20260927 21:36:37.809324 19449 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8109b99760412fbdc911eb18378c1a (DEFAULT_TABLE table=test-table [id=d475b98da63342369d59d9ce58115548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:37.810413 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8109b99760412fbdc911eb18378c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:37.810014 19716 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8109b99760412fbdc911eb18378c1a (DEFAULT_TABLE table=test-table [id=d475b98da63342369d59d9ce58115548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:36:37.811226 19716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8109b99760412fbdc911eb18378c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:36:37.820243 19799 tablet_bootstrap.cc:492] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Bootstrap starting.
I20260927 21:36:37.821379 19800 tablet_bootstrap.cc:492] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Bootstrap starting.
I20260927 21:36:37.823971 19801 tablet_bootstrap.cc:492] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Bootstrap starting.
I20260927 21:36:37.824505 19799 tablet_bootstrap.cc:654] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:37.824507 19800 tablet_bootstrap.cc:654] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:37.825811 19800 log.cc:824] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:37.826381 19799 log.cc:824] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:37.827198 19801 tablet_bootstrap.cc:654] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Neither blocks nor log segments found. Creating new log.
I20260927 21:36:37.828154 19801 log.cc:824] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:37.828440 19800 tablet_bootstrap.cc:492] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: No bootstrap required, opened a new log
I20260927 21:36:37.828718 19800 ts_tablet_manager.cc:1397] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:36:37.830366 19799 tablet_bootstrap.cc:492] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: No bootstrap required, opened a new log
I20260927 21:36:37.830667 19799 ts_tablet_manager.cc:1397] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 21:36:37.831035 19801 tablet_bootstrap.cc:492] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: No bootstrap required, opened a new log
I20260927 21:36:37.831269 19801 ts_tablet_manager.cc:1397] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 21:36:37.835502 19800 raft_consensus.cc:348] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:37.835836 19800 raft_consensus.cc:374] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:37.835958 19800 raft_consensus.cc:729] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59679a36864e494fa4a99de51e0b302a, State: Initialized, Role: FOLLOWER
I20260927 21:36:37.836508 19800 consensus_queue.cc:260] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:37.837472 19648 heartbeater.cc:503] Master 127.3.186.254:46229 was elected leader, sending a full tablet report...
I20260927 21:36:37.838096 19800 ts_tablet_manager.cc:1428] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260927 21:36:37.841040 19801 raft_consensus.cc:348] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:37.841042 19799 raft_consensus.cc:348] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:37.841423 19801 raft_consensus.cc:374] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:37.841423 19799 raft_consensus.cc:374] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:36:37.841531 19801 raft_consensus.cc:729] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687e89e535894539865ea65f0a276287, State: Initialized, Role: FOLLOWER
I20260927 21:36:37.841536 19799 raft_consensus.cc:729] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf994a8535b470b85f0fe962e5cfda6, State: Initialized, Role: FOLLOWER
I20260927 21:36:37.842072 19801 consensus_queue.cc:260] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:37.842058 19799 consensus_queue.cc:260] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:37.843097 19781 heartbeater.cc:503] Master 127.3.186.254:46229 was elected leader, sending a full tablet report...
I20260927 21:36:37.843788 19801 ts_tablet_manager.cc:1428] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 21:36:37.844187 19799 ts_tablet_manager.cc:1428] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260927 21:36:37.958382 19782 tablet.cc:2369] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:36:37.982758 19649 tablet.cc:2369] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:36:38.015807 19515 tablet.cc:2369] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:36:38.062207 19806 raft_consensus.cc:482] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:36:38.062500 19806 raft_consensus.cc:504] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:38.064083 19806 leader_election.cc:288] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117), baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193:33895)
I20260927 21:36:38.069561 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.064620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:56023 (local address 127.3.186.194:46117)
0927 21:36:38.064822 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:38.064828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:38.064847 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:36:38.064924 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:38.064932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:38.065019 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:38.065158 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:38.065735 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.066444 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.067952 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.068710 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.068873 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.068943 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:38.069186 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 21:36:38.069307 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:36:38.069579 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.064379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:46117 (local address 127.3.186.195:56023)
0927 21:36:38.064636 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:38.064657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:38.064774 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:38.065257 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:38.065264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:38.065295 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:38.065569 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:38.065594 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.066600 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.066605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:38.067804 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:38.067817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.068861 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.068866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:38.069036 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.069059 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:36:38.069161 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:36:38.069282 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 21:36:38.070732 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.065148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:33753 (local address 127.3.186.193:33895)
0927 21:36:38.065351 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:38.065359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:38.065854 (+ 495us) server_negotiation.cc:407] Connection header received
0927 21:36:38.066041 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:38.066051 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:38.066106 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:38.066205 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:36:38.066727 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.067408 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.068969 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.069945 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.070099 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.070174 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:38.070470 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 21:36:38.070577 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:36:38.070894 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.064840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33895 (local address 127.3.186.195:33753)
0927 21:36:38.065749 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:38.065763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:38.065899 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:38.066217 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:38.066222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:38.066257 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:36:38.066581 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:38.066589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.067557 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.067563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:38.068818 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:38.068828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.070128 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.070133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:38.070300 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.070348 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:36:38.070441 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:36:38.070589 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":597,"threads_started":1}
I20260927 21:36:38.070698 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8109b99760412fbdc911eb18378c1a" candidate_uuid: "687e89e535894539865ea65f0a276287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59679a36864e494fa4a99de51e0b302a" is_pre_election: true
I20260927 21:36:38.071192 19603 raft_consensus.cc:2454] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687e89e535894539865ea65f0a276287 in term 0.
I20260927 21:36:38.071779 19669 leader_election.cc:302] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59679a36864e494fa4a99de51e0b302a, 687e89e535894539865ea65f0a276287; no voters:
I20260927 21:36:38.071856 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8109b99760412fbdc911eb18378c1a" candidate_uuid: "687e89e535894539865ea65f0a276287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf994a8535b470b85f0fe962e5cfda6" is_pre_election: true
I20260927 21:36:38.072111 19806 raft_consensus.cc:2781] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:36:38.072285 19806 raft_consensus.cc:482] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:36:38.072394 19806 raft_consensus.cc:3037] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:38.072384 19469 raft_consensus.cc:2454] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687e89e535894539865ea65f0a276287 in term 0.
I20260927 21:36:38.075362 19806 raft_consensus.cc:504] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:38.076113 19806 leader_election.cc:288] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [CANDIDATE]: Term 1 election: Requested vote from peers 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117), baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193:33895)
I20260927 21:36:38.076505 19603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8109b99760412fbdc911eb18378c1a" candidate_uuid: "687e89e535894539865ea65f0a276287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59679a36864e494fa4a99de51e0b302a"
I20260927 21:36:38.076735 19603 raft_consensus.cc:3037] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:38.076702 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8109b99760412fbdc911eb18378c1a" candidate_uuid: "687e89e535894539865ea65f0a276287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf994a8535b470b85f0fe962e5cfda6"
I20260927 21:36:38.076900 19469 raft_consensus.cc:3037] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:36:38.080440 19469 raft_consensus.cc:2454] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687e89e535894539865ea65f0a276287 in term 1.
I20260927 21:36:38.080621 19603 raft_consensus.cc:2454] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687e89e535894539865ea65f0a276287 in term 1.
I20260927 21:36:38.080828 19669 leader_election.cc:302] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687e89e535894539865ea65f0a276287, baf994a8535b470b85f0fe962e5cfda6; no voters:
I20260927 21:36:38.081094 19806 raft_consensus.cc:2781] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:36:38.081533 19806 raft_consensus.cc:686] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 1 LEADER]: Becoming Leader. State: Replica: 687e89e535894539865ea65f0a276287, State: Running, Role: LEADER
I20260927 21:36:38.082116 19806 consensus_queue.cc:237] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } }
I20260927 21:36:38.087056 19324 catalog_manager.cc:5495] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 687e89e535894539865ea65f0a276287 (127.3.186.195). New cstate: current_term: 1 leader_uuid: "687e89e535894539865ea65f0a276287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } health_report { overall_health: UNKNOWN } } }
I20260927 21:36:38.148171 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.142056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.3.186.195:44717)
0927 21:36:38.142256 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:38.142263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:38.142284 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:36:38.142379 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:38.142383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:38.142484 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:38.142655 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:36:38.142971 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.144175 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.146003 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.146567 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.146695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.147168 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:36:38.147627 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:36:38.147775 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:38.147831 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 21:36:38.147911 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:36:38.155295 19603 raft_consensus.cc:1261] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Refusing update from remote peer 687e89e535894539865ea65f0a276287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:38.155364 19469 raft_consensus.cc:1261] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Refusing update from remote peer 687e89e535894539865ea65f0a276287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:36:38.156224 19809 consensus_queue.cc:1035] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:38.156586 19806 consensus_queue.cc:1035] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:36:38.167268 19814 mvcc.cc:205] Tried to move back new op lower bound from 7334072312434790400 to 7334072312148762624. Current Snapshot: MvccSnapshot[applied={T|T < 7334072312434790400}]
I20260927 21:36:38.180467 19324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:38.180732 19324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:36:38.184598 19324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Sending DeleteTablet for 3 replicas of tablet 7c8109b99760412fbdc911eb18378c1a
I20260927 21:36:38.185752 19716 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8109b99760412fbdc911eb18378c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:38 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:37916
I20260927 21:36:38.185748 19583 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8109b99760412fbdc911eb18378c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:38 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:54574
I20260927 21:36:38.185926 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19386
I20260927 21:36:38.188072 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8109b99760412fbdc911eb18378c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:36:38 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:37464
I20260927 21:36:38.198206 19821 tablet_replica.cc:330] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: stopping tablet replica
I20260927 21:36:38.198621 19821 raft_consensus.cc:2229] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:36:38.199218 19821 raft_consensus.cc:2258] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:38.202726 19821 ts_tablet_manager.cc:1905] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:38.212508 19824 tablet_replica.cc:330] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: stopping tablet replica
I20260927 21:36:38.212957 19824 raft_consensus.cc:2229] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:38.213490 19824 raft_consensus.cc:2258] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:38.218709 19824 ts_tablet_manager.cc:1905] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:38.219106 19821 ts_tablet_manager.cc:1918] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:38.219272 19821 log.cc:1196] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/7c8109b99760412fbdc911eb18378c1a
I20260927 21:36:38.219807 19821 ts_tablet_manager.cc:1939] T 7c8109b99760412fbdc911eb18378c1a P 687e89e535894539865ea65f0a276287: Deleting consensus metadata
I20260927 21:36:38.221344 19309 catalog_manager.cc:4841] TS 687e89e535894539865ea65f0a276287 (127.3.186.195:44717): tablet 7c8109b99760412fbdc911eb18378c1a (table test-table [id=d475b98da63342369d59d9ce58115548]) successfully deleted
I20260927 21:36:38.225203 19822 tablet_replica.cc:330] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: stopping tablet replica
I20260927 21:36:38.225607 19822 raft_consensus.cc:2229] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:36:38.226071 19822 raft_consensus.cc:2258] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:36:38.226472 19824 ts_tablet_manager.cc:1918] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:38.226675 19824 log.cc:1196] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/7c8109b99760412fbdc911eb18378c1a
I20260927 21:36:38.227118 19824 ts_tablet_manager.cc:1939] T 7c8109b99760412fbdc911eb18378c1a P baf994a8535b470b85f0fe962e5cfda6: Deleting consensus metadata
I20260927 21:36:38.228457 19308 catalog_manager.cc:4841] TS baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193:33895): tablet 7c8109b99760412fbdc911eb18378c1a (table test-table [id=d475b98da63342369d59d9ce58115548]) successfully deleted
Sep 27 21:36:38 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:36:38.230674 19822 ts_tablet_manager.cc:1905] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:36:38.238037 19822 ts_tablet_manager.cc:1918] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:36:38.238183 19822 log.cc:1196] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/7c8109b99760412fbdc911eb18378c1a
I20260927 21:36:38.238703 19822 ts_tablet_manager.cc:1939] T 7c8109b99760412fbdc911eb18378c1a P 59679a36864e494fa4a99de51e0b302a: Deleting consensus metadata
I20260927 21:36:38.239084 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.187160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.3.186.193:33895)
0927 21:36:38.187369 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:38.187377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:38.212025 (+ 24648us) server_negotiation.cc:407] Connection header received
0927 21:36:38.214228 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:38.214237 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:38.214344 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:38.214428 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:38.219818 (+ 5390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.220992 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.223266 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.223918 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.224030 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.231094 (+ 7064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:38.231108 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:38.231123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:38.231159 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:38.235639 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:38.236528 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:38.236537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:38.236540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:38.236679 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:38.237095 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:38.237100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:38.237103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:38.237539 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:38.237817 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:38.238488 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 21:36:38.238703 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:36:38.240365 19509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:38.240497 19308 catalog_manager.cc:4841] TS 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117): tablet 7c8109b99760412fbdc911eb18378c1a (table test-table [id=d475b98da63342369d59d9ce58115548]) successfully deleted
I20260927 21:36:38.343837 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19386
I20260927 21:36:38.361471 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19520
Sep 27 21:36:38 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:36:38.377118 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.362264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.3.186.194:46117)
0927 21:36:38.362465 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:38.362471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:38.362491 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:36:38.362668 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:38.362674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:38.362763 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:38.362862 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:38.363940 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.364972 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.366415 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.367045 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.367168 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.370931 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:38.370947 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:38.370964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:38.371001 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:38.374402 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:38.374988 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:38.374997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:38.375001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:38.375172 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:38.375574 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:38.375579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:38.375582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:38.376014 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:38.376299 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:38.376538 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:36:38.376752 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:36:38.377771 19643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:38.464028 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19520
I20260927 21:36:38.480366 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19653
Sep 27 21:36:38 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:36:38.495915 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:38.481127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.3.186.195:44717)
0927 21:36:38.481293 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:38.481299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:38.481371 (+ 72us) server_negotiation.cc:407] Connection header received
0927 21:36:38.481654 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:38.481660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:38.481741 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:38.481843 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:38.483108 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.483883 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.485183 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:38.485701 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:38.485853 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:38.490003 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:38.490024 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:38.490037 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:38.490070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:38.492901 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:38.493792 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:38.493800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:38.493810 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:38.493971 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:38.494384 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:38.494388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:38.494401 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:38.494713 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:38.495048 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:38.495332 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:36:38.495554 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:36:38.496521 19776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:38.613137 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19653
I20260927 21:36:38.634805 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19292
I20260927 21:36:38.635892 19354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:36:38.738389 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19292
I20260927 21:36:58.758929 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46229
--webserver_interface=127.3.186.254
--webserver_port=38647
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:36:59.066241 19829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:59.066550 19829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:59.066712 19829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:59.073596 19829 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:36:59.073674 19829 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:36:59.073755 19829 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:36:59.073848 19829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:36:59.073936 19829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:59.074034 19829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:59.074110 19829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:36:59.074187 19829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:36:59.083240 19829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=38647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:59.084333 19829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:59.085592 19829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:59.107359 19835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:59.107277 19838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:59.107319 19836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:59.113181 19829 server_base.cc:1034] running on GCE node
I20260927 21:36:59.114075 19829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:59.115531 19829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:59.116775 19829 hybrid_clock.cc:648] HybridClock initialized: now 1790545019116719 us; error 53 us; skew 500 ppm
I20260927 21:36:59.117146 19829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:59 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:59.120549 19829 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:36:59.122179 19829 webserver.cc:460] Webserver started at http://127.3.186.254:38647/ using document root <none> and password file <none>
I20260927 21:36:59.122916 19829 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:59.123067 19829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:59.128690 19829 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 21:36:59.131520 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:59.132741 19829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:36:59.132917 19829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "888c4474e23544ebbf48c164e5a18991"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "53a4be3df34194846eaf3dc1c644708e"
server_key_iv: "b5f4eb58e7170f482e09977ddb3efc76"
server_key_version: "encryptionkey@0"
I20260927 21:36:59.133945 19829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:59.145893 19829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:59.149578 19829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:59.149847 19829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:59.167739 19829 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:46229
I20260927 21:36:59.167784 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:46229 every 8 connection(s)
I20260927 21:36:59.169112 19829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:36:59.175621 19897 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:36:59.178639 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19829
I20260927 21:36:59.178680 19897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap starting.
I20260927 21:36:59.179392 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:33895
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:46229
--webserver_port=36437
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:36:59.201857 19897 log.cc:824] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Log is configured to *not* fsync() on all Append() calls
I20260927 21:36:59.216866 19897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:36:59.217463 19897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap complete.
I20260927 21:36:59.230155 19897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:59.231261 19897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888c4474e23544ebbf48c164e5a18991, State: Initialized, Role: FOLLOWER
I20260927 21:36:59.231942 19897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:59.232210 19897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:36:59.232326 19897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:36:59.232504 19897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:36:59.238226 19897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:59.238837 19897 leader_election.cc:302] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888c4474e23544ebbf48c164e5a18991; no voters:
I20260927 21:36:59.239432 19897 leader_election.cc:288] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:36:59.239544 19901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:36:59.240921 19901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 LEADER]: Becoming Leader. State: Replica: 888c4474e23544ebbf48c164e5a18991, State: Running, Role: LEADER
I20260927 21:36:59.241500 19897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:36:59.241442 19901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:36:59.243695 19903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888c4474e23544ebbf48c164e5a18991. Latest consensus state: current_term: 2 leader_uuid: "888c4474e23544ebbf48c164e5a18991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } } }
I20260927 21:36:59.244094 19903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:59.244650 19902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "888c4474e23544ebbf48c164e5a18991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } } }
I20260927 21:36:59.244971 19902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: This master's current role is: LEADER
I20260927 21:36:59.252308 19907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:36:59.259713 19907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:36:59.261096 19907 catalog_manager.cc:1259] Loaded cluster ID: 3876921eec67431e8cf75afefea7ba53
I20260927 21:36:59.261216 19907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:36:59.264940 19907 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:36:59.266464 19907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Loaded TSK: 0
I20260927 21:36:59.292660 19919 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:36:59.299186 19907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Generated new TSK 1
I20260927 21:36:59.299575 19907 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:36:59.531893 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:59.532167 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:59.532313 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:59.539497 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:59.539630 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:59.539783 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:36:59.549070 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:33895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=36437
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:59.550232 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:59.551430 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:59.563848 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:59.563858 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:59.564997 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:59.565440 19899 server_base.cc:1034] running on GCE node
I20260927 21:36:59.566006 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:59.566872 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:59.568095 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790545019568042 us; error 46 us; skew 500 ppm
I20260927 21:36:59.568391 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:59 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:59.571378 19899 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:36:59.572841 19899 webserver.cc:460] Webserver started at http://127.3.186.193:36437/ using document root <none> and password file <none>
I20260927 21:36:59.573546 19899 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:59.573689 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:36:59.578730 19899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:36:59.581254 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:59.582571 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:36:59.582731 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "baf994a8535b470b85f0fe962e5cfda6"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "9173fecedf4468bd3e3bce161e56d945"
server_key_iv: "391201c258f3625645f408f64fe7cf22"
server_key_version: "encryptionkey@0"
I20260927 21:36:59.583694 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:36:59.599478 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:36:59.603565 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:36:59.603797 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:36:59.604637 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:36:59.605839 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:36:59.605930 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:59.606005 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:36:59.606062 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:36:59.638161 19899 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:33895
I20260927 21:36:59.638262 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:33895 every 8 connection(s)
I20260927 21:36:59.639492 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:36:59.644385 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 19899
I20260927 21:36:59.644972 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:46117
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:46229
--webserver_port=40333
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:36:59 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:36:59.661391 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:59.642422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:58643)
0927 21:36:59.643118 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 21:36:59.643152 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:36:59.644877 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:36:59.646351 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:36:59.646361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:36:59.646929 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:36:59.647426 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:59.647452 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:59.648958 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:59.648964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:59.650104 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:36:59.650115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:36:59.651092 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:36:59.651097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:36:59.651268 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:59.651996 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:36:59.652016 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:36:59.654329 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:36:59.657558 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:59.657566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:59.657578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:59.657953 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:59.658351 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:36:59.658358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:36:59.658362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:36:59.658546 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:36:59.659211 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:36:59.659217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:36:59.659536 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 21:36:59.659776 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 21:36:59.660246 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":294,"threads_started":1}
I20260927 21:36:59.661392 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:36:59.642109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:58643 (local address 127.3.186.254:46229)
0927 21:36:59.642647 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 21:36:59.642658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:36:59.643271 (+ 613us) server_negotiation.cc:407] Connection header received
0927 21:36:59.645458 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:36:59.645470 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:36:59.645842 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:36:59.646189 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:36:59.647623 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:59.648814 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:59.650236 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:36:59.650884 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:36:59.651078 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:36:59.654589 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:36:59.654596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:36:59.654609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:36:59.654640 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:36:59.657388 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:59.658092 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:59.658098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:59.658102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:59.658244 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:36:59.658668 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:36:59.658672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:36:59.658674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:36:59.659081 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:36:59.659277 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 21:36:59.659984 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 21:36:59.660331 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260927 21:36:59.663918 20048 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46229
I20260927 21:36:59.664196 20048 heartbeater.cc:464] Registering TS with master...
I20260927 21:36:59.664835 20048 heartbeater.cc:511] Master 127.3.186.254:46229 requested a full tablet report, sending...
I20260927 21:36:59.666810 19862 ts_manager.cc:194] Registered new tserver with Master: baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193:33895)
I20260927 21:36:59.668206 19862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:58643
W20260927 21:36:59.955327 20052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:36:59.955602 20052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:36:59.955765 20052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:36:59.963228 20052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:36:59.963407 20052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:36:59.963583 20052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:36:59.973296 20052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=40333
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:36:59.974350 20052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:36:59.975492 20052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:36:59.987643 20060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:59.987673 20057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:36:59.987676 20058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:36:59.988305 20052 server_base.cc:1034] running on GCE node
I20260927 21:36:59.988818 20052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:36:59.989701 20052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:36:59.990906 20052 hybrid_clock.cc:648] HybridClock initialized: now 1790545019990875 us; error 51 us; skew 500 ppm
I20260927 21:36:59.991178 20052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:36:59 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:36:59.993953 20052 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:36:59.995137 20052 webserver.cc:460] Webserver started at http://127.3.186.194:40333/ using document root <none> and password file <none>
I20260927 21:36:59.995713 20052 fs_manager.cc:359] Metadata directory not provided
I20260927 21:36:59.995846 20052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:00.000733 20052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:37:00.003242 20067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:00.004406 20052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:37:00.004527 20052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "59679a36864e494fa4a99de51e0b302a"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "5d325ad9c35a87ed7cd7666416c3e103"
server_key_iv: "e9e5e7add2b7626971fd2dda67a1c7cc"
server_key_version: "encryptionkey@0"
I20260927 21:37:00.005338 20052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:00.028239 20052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:00.031801 20052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:00.032018 20052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:00.032810 20052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:00.033950 20052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:00.034029 20052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:00.034104 20052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:00.034157 20052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:00.066915 20052 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:46117
I20260927 21:37:00.067028 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:46117 every 8 connection(s)
I20260927 21:37:00.068327 20052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:00.076663 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20052
I20260927 21:37:00.077270 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:44717
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:46229
--webserver_port=40837
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:37:00 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545019, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:00.087790 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.070583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:47293 (local address 127.3.186.254:46229)
0927 21:37:00.070754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.070759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.071805 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 21:37:00.073644 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.073649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.073739 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.073847 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:00.075261 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.075956 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.077616 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.078021 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.078148 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.081869 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:00.081873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:00.081877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:00.081909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:00.084767 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.085445 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.085453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.085459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.085623 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.086029 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.086036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.086043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.086464 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:00.086617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.087196 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.087449 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:37:00.088586 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.071015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:47293)
0927 21:37:00.071647 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.071683 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.073336 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.074079 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.074086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.074537 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:00.075105 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.075122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.076115 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.076121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.077423 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.077441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.078245 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.078252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.078457 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.079094 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:00.079117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:00.081640 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:00.084906 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:00.084912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:00.084921 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:00.085315 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:00.085738 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:00.085742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:00.085744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:00.085901 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:00.086580 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:00.086585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:00.086851 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.087102 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.087551 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":256,"threads_started":1}
I20260927 21:37:00.089986 20180 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46229
I20260927 21:37:00.090286 20180 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:00.091061 20180 heartbeater.cc:511] Master 127.3.186.254:46229 requested a full tablet report, sending...
I20260927 21:37:00.092555 19862 ts_manager.cc:194] Registered new tserver with Master: 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117)
I20260927 21:37:00.093796 19862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:47293
W20260927 21:37:00.387398 20183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:00.387631 20183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:00.387714 20183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:00.394414 20183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:00.394502 20183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:00.394600 20183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:00.403563 20183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:44717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=40837
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:00.404697 20183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:00.405931 20183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:00.418287 20191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:00.418618 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:00.418377 20189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:00.419471 20183 server_base.cc:1034] running on GCE node
I20260927 21:37:00.420064 20183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:00.420943 20183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:00.422158 20183 hybrid_clock.cc:648] HybridClock initialized: now 1790545020422136 us; error 60 us; skew 500 ppm
I20260927 21:37:00.422511 20183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:00 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:00.425383 20183 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:37:00.426813 20183 webserver.cc:460] Webserver started at http://127.3.186.195:40837/ using document root <none> and password file <none>
I20260927 21:37:00.427456 20183 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:00.427635 20183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:00.432453 20183 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:37:00.435001 20198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:00.436234 20183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:37:00.436437 20183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "687e89e535894539865ea65f0a276287"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "8190892c134bd5e852afc21c03e4a45d"
server_key_iv: "e47296414f83403f8627e6ef86cda7cd"
server_key_version: "encryptionkey@0"
I20260927 21:37:00.437445 20183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:00.456866 20183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:00.460261 20183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:00.460497 20183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:00.461277 20183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:00.462499 20183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:00.462618 20183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:00.462746 20183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:00.462837 20183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:00.493448 20183 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:44717
I20260927 21:37:00.493567 20310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:44717 every 8 connection(s)
I20260927 21:37:00.494722 20183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:00.501144 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20183
Sep 27 21:37:00 dist-test-slave-pqr8 krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545020, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:00.516273 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.496960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:60027 (local address 127.3.186.254:46229)
0927 21:37:00.497165 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.497172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.498228 (+ 1056us) server_negotiation.cc:407] Connection header received
0927 21:37:00.499966 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.499970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.500030 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.500112 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:00.501772 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.502943 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.505194 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.506062 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.506180 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.509929 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:00.509935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:00.509940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:00.509981 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:00.512992 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.513902 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.513906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.513908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.514034 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.514542 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.514546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.514548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.514808 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:00.514927 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.515689 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.515886 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":71}
I20260927 21:37:00.517062 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.497370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:60027)
0927 21:37:00.498071 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.498113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.499726 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.500297 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.500304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.500828 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:00.501587 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.501612 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.503117 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.503124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.505066 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.505083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.506261 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.506266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.506464 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.507134 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:00.507168 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:00.509712 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:00.513120 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:00.513130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:00.513144 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:00.513780 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:00.514149 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:00.514156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:00.514163 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:00.514410 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:00.514945 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:00.514953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:00.515320 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.515617 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.516057 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":328,"threads_started":1}
I20260927 21:37:00.518478 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.502118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.3.186.254:46229)
0927 21:37:00.502819 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.502824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.502841 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:37:00.502943 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.502948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.503021 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.503122 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:00.505194 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.506432 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.508275 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.508711 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.508797 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.512658 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:00.512663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:00.512665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:00.512708 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:00.515782 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.516541 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.516548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.516554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.516661 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.517045 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.517051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.517054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.517349 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:00.517657 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.517950 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.518159 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":592,"thread_start_us":471,"threads_started":1}
I20260927 21:37:00.518600 20311 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46229
I20260927 21:37:00.518893 20311 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:00.519652 20311 heartbeater.cc:511] Master 127.3.186.254:46229 requested a full tablet report, sending...
I20260927 21:37:00.521064 19862 ts_manager.cc:194] Registered new tserver with Master: 687e89e535894539865ea65f0a276287 (127.3.186.195:44717)
I20260927 21:37:00.521863 19862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:60027
I20260927 21:37:00.532605 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:00.538532 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.533352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.3.186.254:46229)
0927 21:37:00.533577 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.533585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.533961 (+ 376us) server_negotiation.cc:407] Connection header received
0927 21:37:00.534124 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.534127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.534231 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.534336 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:00.534749 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.535420 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.536954 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.537329 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.537466 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.537929 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:00.538092 (+ 163us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 21:37:00.538316 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42368: token expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 21:37:00.538683 20315 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42368: token expired
W20260927 21:37:00.539183 19785 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.186.254:46229)
I20260927 21:37:00.550264 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.539653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.3.186.254:46229)
0927 21:37:00.539801 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.539804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.539837 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:37:00.539905 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.539909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.539955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.540024 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:00.541451 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.542249 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.543476 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.543808 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.543884 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.546104 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:00.546108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:00.546111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:00.546134 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:00.548221 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.548750 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.548753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.548755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.548856 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:00.549131 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:00.549134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:00.549136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:00.549369 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:00.549603 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.549834 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.549978 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:37:00.552986 19862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:00.559902 19862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:00.581816 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.570120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33895 (local address 127.0.0.1:57258)
0927 21:37:00.570897 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.570931 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.571073 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.572238 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.572247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.572284 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:00.572713 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.572727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.574951 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.574958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.577972 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.577991 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.580537 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.580547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.580840 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.580905 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.581052 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.581267 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":283,"threads_started":1}
I20260927 21:37:00.583489 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.570270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.3.186.193:33895)
0927 21:37:00.570948 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.570958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.570989 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:37:00.571548 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.571557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.571784 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.571978 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:00.572939 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.574756 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.578179 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.579979 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.580121 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.580375 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.582393 (+ 2018us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.582615 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":407,"threads_started":1}
I20260927 21:37:00.589138 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.571101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:44717 (local address 127.0.0.1:50682)
0927 21:37:00.580226 (+ 9125us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.580247 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.580379 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.581677 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.581684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.581722 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:00.582218 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.582232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.584503 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.584511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.586672 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.586685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.588376 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.588382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.588545 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.588577 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.588667 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.588779 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9001,"mutex_wait_us":23,"thread_start_us":8881,"threads_started":1}
I20260927 21:37:00.589190 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.571648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.3.186.195:44717)
0927 21:37:00.580690 (+ 9042us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.580701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.580737 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:37:00.580843 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.580853 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.581129 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.581350 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:00.582433 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.584313 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.586855 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.588169 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.588298 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.588506 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.588740 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.588891 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8834,"thread_start_us":8681,"threads_started":1}
I20260927 21:37:00.588876 19983 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6f2ef5329348f48bbab0651a5127e9 (DEFAULT_TABLE table=test-table [id=5d0aa9abd5354e188a3afd7ea175bba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:00.590286 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6f2ef5329348f48bbab0651a5127e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:00.591571 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.571116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:46117 (local address 127.0.0.1:57610)
0927 21:37:00.582051 (+ 10935us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.582066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.582177 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.585447 (+ 3270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.585452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.585476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:00.585719 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.585730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.587254 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.587259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.589072 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.589090 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.590854 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.590859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.591021 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.591059 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.591160 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.591278 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10858,"mutex_wait_us":1,"thread_start_us":13794,"threads_started":1}
I20260927 21:37:00.591616 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.571737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.3.186.194:46117)
0927 21:37:00.584848 (+ 13111us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.584857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.584891 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:37:00.585013 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.585022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.585255 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.585450 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:00.585897 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.586998 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.589359 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.590693 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.590829 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.591002 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.591205 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.591338 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12950,"thread_start_us":12815,"threads_started":1}
I20260927 21:37:00.592881 20246 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6f2ef5329348f48bbab0651a5127e9 (DEFAULT_TABLE table=test-table [id=5d0aa9abd5354e188a3afd7ea175bba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:00.594110 20246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6f2ef5329348f48bbab0651a5127e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:00.594463 20115 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6f2ef5329348f48bbab0651a5127e9 (DEFAULT_TABLE table=test-table [id=5d0aa9abd5354e188a3afd7ea175bba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:00.595278 20115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6f2ef5329348f48bbab0651a5127e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:00.605218 20327 tablet_bootstrap.cc:492] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Bootstrap starting.
I20260927 21:37:00.606909 20328 tablet_bootstrap.cc:492] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Bootstrap starting.
I20260927 21:37:00.608238 20329 tablet_bootstrap.cc:492] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Bootstrap starting.
I20260927 21:37:00.610383 20328 tablet_bootstrap.cc:654] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:00.610630 20327 tablet_bootstrap.cc:654] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:00.611377 20328 log.cc:824] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:00.611500 20329 tablet_bootstrap.cc:654] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:00.611780 20327 log.cc:824] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:00.612653 20329 log.cc:824] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:00.614152 20328 tablet_bootstrap.cc:492] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: No bootstrap required, opened a new log
I20260927 21:37:00.614413 20328 ts_tablet_manager.cc:1397] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:37:00.615131 20327 tablet_bootstrap.cc:492] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: No bootstrap required, opened a new log
I20260927 21:37:00.615379 20327 ts_tablet_manager.cc:1397] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 21:37:00.615584 20329 tablet_bootstrap.cc:492] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: No bootstrap required, opened a new log
I20260927 21:37:00.615842 20329 ts_tablet_manager.cc:1397] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:37:00.622273 20327 raft_consensus.cc:348] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.622601 20327 raft_consensus.cc:374] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:00.622722 20327 raft_consensus.cc:729] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf994a8535b470b85f0fe962e5cfda6, State: Initialized, Role: FOLLOWER
I20260927 21:37:00.623270 20327 consensus_queue.cc:260] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.624281 20048 heartbeater.cc:503] Master 127.3.186.254:46229 was elected leader, sending a full tablet report...
I20260927 21:37:00.624722 20327 ts_tablet_manager.cc:1428] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260927 21:37:00.625207 20328 raft_consensus.cc:348] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.625578 20328 raft_consensus.cc:374] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:00.625761 20328 raft_consensus.cc:729] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59679a36864e494fa4a99de51e0b302a, State: Initialized, Role: FOLLOWER
I20260927 21:37:00.625728 20329 raft_consensus.cc:348] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.626097 20329 raft_consensus.cc:374] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:00.626212 20329 raft_consensus.cc:729] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687e89e535894539865ea65f0a276287, State: Initialized, Role: FOLLOWER
I20260927 21:37:00.626395 20328 consensus_queue.cc:260] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.626717 20329 consensus_queue.cc:260] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.627521 20180 heartbeater.cc:503] Master 127.3.186.254:46229 was elected leader, sending a full tablet report...
I20260927 21:37:00.628083 20328 ts_tablet_manager.cc:1428] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260927 21:37:00.628916 20311 heartbeater.cc:503] Master 127.3.186.254:46229 was elected leader, sending a full tablet report...
I20260927 21:37:00.629587 20329 ts_tablet_manager.cc:1428] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
W20260927 21:37:00.641362 20049 tablet.cc:2369] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:37:00.746219 20312 tablet.cc:2369] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:37:00.820310 20181 tablet.cc:2369] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:00.930199 20333 raft_consensus.cc:482] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:00.930580 20333 raft_consensus.cc:504] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.932164 20333 leader_election.cc:288] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117), 687e89e535894539865ea65f0a276287 (127.3.186.195:44717)
I20260927 21:37:00.940498 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.932545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:48001 (local address 127.3.186.194:46117)
0927 21:37:00.932755 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.932761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.933739 (+ 978us) server_negotiation.cc:407] Connection header received
0927 21:37:00.933921 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.933926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.933988 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.934052 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:00.934674 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.935669 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.938422 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.939522 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.939666 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.939732 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.940117 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.940280 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:37:00.940465 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.932546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:37355 (local address 127.3.186.195:44717)
0927 21:37:00.932755 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:00.932761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:00.933278 (+ 517us) server_negotiation.cc:407] Connection header received
0927 21:37:00.933599 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:00.933604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:00.933664 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:00.933773 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:00.934383 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.935581 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.938422 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.939521 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.939666 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.939731 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:00.940117 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 21:37:00.940232 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:37:00.940624 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.932397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:46117 (local address 127.3.186.193:48001)
0927 21:37:00.933642 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.933656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.933796 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.934098 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.934101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.934130 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:00.934536 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.934549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.935806 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.935815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.938282 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.938290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.939737 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.939747 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.939954 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.939995 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.940084 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.940232 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":173,"thread_start_us":878,"threads_started":1}
I20260927 21:37:00.940951 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:00.932397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:44717 (local address 127.3.186.193:37355)
0927 21:37:00.933171 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:00.933186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:00.933410 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:00.933805 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:00.933809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:00.933832 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:00.934205 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.934221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.935700 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.935704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.938277 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:00.938290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:00.939761 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:00.939765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:00.939948 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:00.940004 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:37:00.940084 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:37:00.940232 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":215,"thread_start_us":514,"threads_started":1}
I20260927 21:37:00.941587 20135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f2ef5329348f48bbab0651a5127e9" candidate_uuid: "baf994a8535b470b85f0fe962e5cfda6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59679a36864e494fa4a99de51e0b302a" is_pre_election: true
I20260927 21:37:00.941830 20266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f2ef5329348f48bbab0651a5127e9" candidate_uuid: "baf994a8535b470b85f0fe962e5cfda6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687e89e535894539865ea65f0a276287" is_pre_election: true
I20260927 21:37:00.942097 20135 raft_consensus.cc:2454] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf994a8535b470b85f0fe962e5cfda6 in term 0.
I20260927 21:37:00.942330 20266 raft_consensus.cc:2454] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baf994a8535b470b85f0fe962e5cfda6 in term 0.
I20260927 21:37:00.942850 19936 leader_election.cc:302] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59679a36864e494fa4a99de51e0b302a, baf994a8535b470b85f0fe962e5cfda6; no voters:
I20260927 21:37:00.943279 20333 raft_consensus.cc:2781] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:00.943440 20333 raft_consensus.cc:482] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:00.943549 20333 raft_consensus.cc:3037] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:00.945591 20333 raft_consensus.cc:504] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.946131 20333 leader_election.cc:288] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [CANDIDATE]: Term 1 election: Requested vote from peers 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117), 687e89e535894539865ea65f0a276287 (127.3.186.195:44717)
I20260927 21:37:00.946609 20266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f2ef5329348f48bbab0651a5127e9" candidate_uuid: "baf994a8535b470b85f0fe962e5cfda6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687e89e535894539865ea65f0a276287"
I20260927 21:37:00.946609 20135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6f2ef5329348f48bbab0651a5127e9" candidate_uuid: "baf994a8535b470b85f0fe962e5cfda6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59679a36864e494fa4a99de51e0b302a"
I20260927 21:37:00.946803 20266 raft_consensus.cc:3037] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:00.946803 20135 raft_consensus.cc:3037] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:00.949759 20266 raft_consensus.cc:2454] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf994a8535b470b85f0fe962e5cfda6 in term 1.
I20260927 21:37:00.949786 20135 raft_consensus.cc:2454] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baf994a8535b470b85f0fe962e5cfda6 in term 1.
I20260927 21:37:00.950109 19936 leader_election.cc:302] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59679a36864e494fa4a99de51e0b302a, baf994a8535b470b85f0fe962e5cfda6; no voters:
I20260927 21:37:00.950404 20333 raft_consensus.cc:2781] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:00.950812 20333 raft_consensus.cc:686] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 1 LEADER]: Becoming Leader. State: Replica: baf994a8535b470b85f0fe962e5cfda6, State: Running, Role: LEADER
I20260927 21:37:00.951313 20333 consensus_queue.cc:237] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } }
I20260927 21:37:00.955677 19861 catalog_manager.cc:5495] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 reported cstate change: term changed from 0 to 1, leader changed from <none> to baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "baf994a8535b470b85f0fe962e5cfda6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf994a8535b470b85f0fe962e5cfda6" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 } health_report { overall_health: UNKNOWN } } }
I20260927 21:37:01.029345 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:01.024389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.3.186.193:33895)
0927 21:37:01.024568 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:01.024573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:01.024591 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:37:01.024798 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:01.024801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:01.024867 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:01.025005 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:01.025331 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.026099 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.027410 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.027810 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.027936 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:01.028389 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:01.028830 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:01.028982 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:01.029042 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:37:01.029110 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:37:01.037199 20266 raft_consensus.cc:1261] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Refusing update from remote peer baf994a8535b470b85f0fe962e5cfda6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:01.037201 20135 raft_consensus.cc:1261] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Refusing update from remote peer baf994a8535b470b85f0fe962e5cfda6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:01.038255 20338 consensus_queue.cc:1035] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59679a36864e494fa4a99de51e0b302a" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:01.039479 20333 consensus_queue.cc:1035] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687e89e535894539865ea65f0a276287" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:01.048563 20341 mvcc.cc:205] Tried to move back new op lower bound from 7334072406158434304 to 7334072405820092416. Current Snapshot: MvccSnapshot[applied={T|T < 7334072406158434304}]
I20260927 21:37:01.062767 19862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:01.062968 19862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:01.066253 19862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Sending DeleteTablet for 3 replicas of tablet 7c6f2ef5329348f48bbab0651a5127e9
I20260927 21:37:01.067395 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f2ef5329348f48bbab0651a5127e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:01 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:57258
I20260927 21:37:01.067653 20246 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f2ef5329348f48bbab0651a5127e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:01 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:50682
I20260927 21:37:01.068269 20115 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6f2ef5329348f48bbab0651a5127e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:01 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:57610
I20260927 21:37:01.074965 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19899
I20260927 21:37:01.077836 20353 tablet_replica.cc:330] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: stopping tablet replica
I20260927 21:37:01.078763 20353 raft_consensus.cc:2229] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:01.079668 20353 raft_consensus.cc:2258] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:01.081913 20353 ts_tablet_manager.cc:1905] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:01.085767 20351 tablet_replica.cc:330] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: stopping tablet replica
I20260927 21:37:01.086131 20351 raft_consensus.cc:2229] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:01.086728 20351 raft_consensus.cc:2258] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:01.090380 20353 ts_tablet_manager.cc:1918] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:01.090576 20353 log.cc:1196] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/7c6f2ef5329348f48bbab0651a5127e9
I20260927 21:37:01.091109 20351 ts_tablet_manager.cc:1905] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:01.091180 20353 ts_tablet_manager.cc:1939] T 7c6f2ef5329348f48bbab0651a5127e9 P 59679a36864e494fa4a99de51e0b302a: Deleting consensus metadata
I20260927 21:37:01.092875 19846 catalog_manager.cc:4841] TS 59679a36864e494fa4a99de51e0b302a (127.3.186.194:46117): tablet 7c6f2ef5329348f48bbab0651a5127e9 (table test-table [id=5d0aa9abd5354e188a3afd7ea175bba3]) successfully deleted
I20260927 21:37:01.097776 20352 tablet_replica.cc:330] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: stopping tablet replica
I20260927 21:37:01.097967 20351 ts_tablet_manager.cc:1918] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:01.098130 20351 log.cc:1196] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/7c6f2ef5329348f48bbab0651a5127e9
I20260927 21:37:01.098147 20352 raft_consensus.cc:2229] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:01.098582 20352 raft_consensus.cc:2258] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:01.098609 20351 ts_tablet_manager.cc:1939] T 7c6f2ef5329348f48bbab0651a5127e9 P baf994a8535b470b85f0fe962e5cfda6: Deleting consensus metadata
I20260927 21:37:01.100099 19846 catalog_manager.cc:4841] TS baf994a8535b470b85f0fe962e5cfda6 (127.3.186.193:33895): tablet 7c6f2ef5329348f48bbab0651a5127e9 (table test-table [id=5d0aa9abd5354e188a3afd7ea175bba3]) successfully deleted
I20260927 21:37:01.102952 20352 ts_tablet_manager.cc:1905] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:01.108918 20352 ts_tablet_manager.cc:1918] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:01.109061 20352 log.cc:1196] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/7c6f2ef5329348f48bbab0651a5127e9
I20260927 21:37:01.109591 20352 ts_tablet_manager.cc:1939] T 7c6f2ef5329348f48bbab0651a5127e9 P 687e89e535894539865ea65f0a276287: Deleting consensus metadata
I20260927 21:37:01.111003 19847 catalog_manager.cc:4841] TS 687e89e535894539865ea65f0a276287 (127.3.186.195:44717): tablet 7c6f2ef5329348f48bbab0651a5127e9 (table test-table [id=5d0aa9abd5354e188a3afd7ea175bba3]) successfully deleted
I20260927 21:37:01.112860 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:01.076066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.3.186.193:33895)
0927 21:37:01.076223 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:01.076228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:01.094715 (+ 18487us) server_negotiation.cc:407] Connection header received
0927 21:37:01.095092 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:01.095097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:01.095210 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:01.095289 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:01.098932 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.100136 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.102025 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.102658 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.102779 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:01.106144 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:01.106157 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:01.106173 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:01.106212 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:01.110115 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:01.110963 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:01.110973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:01.110978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:01.111146 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:01.111548 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:01.111551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:01.111553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:01.111881 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:01.112113 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:01.112440 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 21:37:01.112578 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:37:01.113493 20043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:01.211897 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19899
I20260927 21:37:01.230335 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 20052
I20260927 21:37:01.243607 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:01.231047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.3.186.194:46117)
0927 21:37:01.231223 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:01.231231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:01.231310 (+ 79us) server_negotiation.cc:407] Connection header received
0927 21:37:01.231509 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:01.231514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:01.231587 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:01.231700 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:01.233119 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.233938 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.235304 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.235720 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.235869 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:01.238497 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:01.238508 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:01.238520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:01.238548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:01.241078 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:01.241768 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:01.241775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:01.241778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:01.241967 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:01.242344 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:01.242348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:01.242350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:01.242676 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:01.242983 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:01.243200 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 21:37:01.243358 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:37:01.244184 20175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:01.328253 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 20052
I20260927 21:37:01.345134 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 20183
I20260927 21:37:01.358254 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:01.345905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.3.186.195:44717)
0927 21:37:01.346107 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:01.346113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:01.346131 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:37:01.346302 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:01.346312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:01.346380 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:01.346487 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:01.347787 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.348489 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.349958 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:01.350346 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:01.350478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:01.352979 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:01.352991 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:01.353007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:01.353039 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:01.355536 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:01.356241 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:01.356256 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:01.356258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:01.356389 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:01.356748 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:01.356757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:01.356759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:01.357104 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:01.357383 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:01.357731 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 21:37:01.357947 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:37:01.358860 20306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:01.442623 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 20183
I20260927 21:37:01.458963 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 19829
I20260927 21:37:01.460021 19892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:01.566629 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 19829
I20260927 21:37:21.587699 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46229
--webserver_interface=127.3.186.254
--webserver_port=38647
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:21.901748 20359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:21.901973 20359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:21.902127 20359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:21.908927 20359 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:37:21.909008 20359 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:37:21.909092 20359 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:37:21.909193 20359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:21.909281 20359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:21.909397 20359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:21.909475 20359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:21.909572 20359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:21.918592 20359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=38647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:21.919588 20359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:21.920804 20359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:21.931734 20364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:21.931780 20367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:21.931813 20365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:21.932706 20359 server_base.cc:1034] running on GCE node
I20260927 21:37:21.933411 20359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:21.934617 20359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:21.935889 20359 hybrid_clock.cc:648] HybridClock initialized: now 1790545041935828 us; error 34 us; skew 500 ppm
I20260927 21:37:21.936189 20359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:21 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:21.945188 20359 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:37:21.946550 20359 webserver.cc:460] Webserver started at http://127.3.186.254:38647/ using document root <none> and password file <none>
I20260927 21:37:21.947121 20359 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:21.947204 20359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:21.952085 20359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:37:21.954487 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:21.955598 20359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:37:21.955782 20359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "888c4474e23544ebbf48c164e5a18991"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "53a4be3df34194846eaf3dc1c644708e"
server_key_iv: "b5f4eb58e7170f482e09977ddb3efc76"
server_key_version: "encryptionkey@0"
I20260927 21:37:21.957063 20359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:21.970135 20359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:21.973456 20359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:21.973732 20359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:21.991185 20359 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:46229
I20260927 21:37:21.991255 20425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:46229 every 8 connection(s)
I20260927 21:37:21.992435 20359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:21.998119 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20359
I20260927 21:37:21.999060 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:33895
--local_ip_for_outbound_sockets=127.3.186.193
--tserver_master_addrs=127.3.186.254:46229
--webserver_port=36437
--webserver_interface=127.3.186.193
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:37:21.999109 20426 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:37:22.003846 20426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap starting.
I20260927 21:37:22.024158 20426 log.cc:824] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:22.040059 20426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:37:22.040603 20426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Bootstrap complete.
I20260927 21:37:22.049403 20426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:37:22.050315 20426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888c4474e23544ebbf48c164e5a18991, State: Initialized, Role: FOLLOWER
I20260927 21:37:22.050870 20426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:37:22.051091 20426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:22.051231 20426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:22.051414 20426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:37:22.054939 20426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:37:22.055445 20426 leader_election.cc:302] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888c4474e23544ebbf48c164e5a18991; no voters:
I20260927 21:37:22.056087 20426 leader_election.cc:288] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:37:22.056216 20430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:37:22.057538 20430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [term 3 LEADER]: Becoming Leader. State: Replica: 888c4474e23544ebbf48c164e5a18991, State: Running, Role: LEADER
I20260927 21:37:22.058086 20430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } }
I20260927 21:37:22.058302 20426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:22.060485 20431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "888c4474e23544ebbf48c164e5a18991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } } }
I20260927 21:37:22.060865 20431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:22.061156 20432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888c4474e23544ebbf48c164e5a18991. Latest consensus state: current_term: 3 leader_uuid: "888c4474e23544ebbf48c164e5a18991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888c4474e23544ebbf48c164e5a18991" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46229 } } }
I20260927 21:37:22.061642 20432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:22.063843 20435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:22.073473 20435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:22.075266 20435 catalog_manager.cc:1259] Loaded cluster ID: 3876921eec67431e8cf75afefea7ba53
I20260927 21:37:22.075376 20435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:22.080583 20435 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:22.082528 20435 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Loaded TSK: 0
I20260927 21:37:22.082685 20435 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 888c4474e23544ebbf48c164e5a18991: Loaded TSK: 1
F20260927 21:37:22.084028 20435 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb1b3ba29ad google::LogMessage::Fail() at ??:0
@ 0x7fb1b3ba588a google::LogMessage::SendToLog() at ??:0
@ 0x7fb1b3ba24de google::LogMessage::Flush() at ??:0
@ 0x7fb1b3ba3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb1b79067d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb1b78f573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb1c95d11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260927 21:37:22.126861 20450 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fb1c95d0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb1c965c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb1c965c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb1c96a65e9 std::function<>::operator()() at ??:0
@ 0x7fb1c95d4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb1c95d2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb1c965a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb1c965a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1b67ee345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb1b67f191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb1b67f1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1b67cceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb1b9aa66db start_thread at ??:0
@ 0x7fb1b2a3c61f clone at ??:0
W20260927 21:37:22.393489 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:22.393833 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:22.394011 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:22.401485 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:22.401654 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:22.401819 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:22.410898 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:33895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=36437
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:22.411988 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:22.413200 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:22.425592 20476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:22.425635 20474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:22.426060 20473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:22.426712 20428 server_base.cc:1034] running on GCE node
I20260927 21:37:22.427304 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:22.428143 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:22.429364 20428 hybrid_clock.cc:648] HybridClock initialized: now 1790545042429324 us; error 57 us; skew 500 ppm
I20260927 21:37:22.429785 20428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:22 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:22.432631 20428 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:37:22.434046 20428 webserver.cc:460] Webserver started at http://127.3.186.193:36437/ using document root <none> and password file <none>
I20260927 21:37:22.434653 20428 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:22.434804 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:22.439687 20428 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:37:22.442203 20483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:22.443351 20428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:22.443513 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "baf994a8535b470b85f0fe962e5cfda6"
format_stamp: "Formatted at 2026-09-27 21:36:36 on dist-test-slave-pqr8"
server_key: "9173fecedf4468bd3e3bce161e56d945"
server_key_iv: "391201c258f3625645f408f64fe7cf22"
server_key_version: "encryptionkey@0"
I20260927 21:37:22.444407 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:22.473839 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:22.477344 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:22.477631 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:22.478471 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:22.479720 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:22.479843 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:22.479971 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:22.480072 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:22.512147 20428 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:33895
I20260927 21:37:22.512241 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:33895 every 8 connection(s)
I20260927 21:37:22.513451 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:22.517763 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:22.516075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:40549)
0927 21:37:22.516729 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:22.516956 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":267,"threads_started":1}
W20260927 21:37:22.518327 20596 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:46229 (0 consecutive failures): Network error: Failed to ping master at 127.3.186.254:46229: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
I20260927 21:37:22.519013 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:22.518762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:37351)
0927 21:37:22.518871 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:22.518925 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 21:37:22.519780 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:22.519530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:40663)
0927 21:37:22.519630 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:22.519691 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 21:37:22.519989 20596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:37:22.523603 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20428
I20260927 21:37:22.524154 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:46117
--local_ip_for_outbound_sockets=127.3.186.194
--tserver_master_addrs=127.3.186.254:46229
--webserver_port=40333
--webserver_interface=127.3.186.194
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:22.832583 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:22.832864 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:22.833043 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:22.840052 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:22.840186 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:22.840348 20599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:22.849478 20599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=40333
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:22.850677 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:22.851868 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:22.864400 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:22.864390 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:22.864429 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:22.865314 20599 server_base.cc:1034] running on GCE node
I20260927 21:37:22.865872 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:22.866709 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:22.867895 20599 hybrid_clock.cc:648] HybridClock initialized: now 1790545042867860 us; error 38 us; skew 500 ppm
I20260927 21:37:22.868161 20599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:22 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:22.871078 20599 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:37:22.872470 20599 webserver.cc:460] Webserver started at http://127.3.186.194:40333/ using document root <none> and password file <none>
I20260927 21:37:22.873070 20599 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:22.873204 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:22.877978 20599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:37:22.880362 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:22.881608 20599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:37:22.881768 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "59679a36864e494fa4a99de51e0b302a"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "5d325ad9c35a87ed7cd7666416c3e103"
server_key_iv: "e9e5e7add2b7626971fd2dda67a1c7cc"
server_key_version: "encryptionkey@0"
I20260927 21:37:22.882642 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:22.898887 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:22.902469 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:22.902715 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:22.903600 20599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:22.904805 20599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:22.904928 20599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:22.905066 20599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:22.905153 20599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:22.936944 20599 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:46117
I20260927 21:37:22.937004 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:46117 every 8 connection(s)
I20260927 21:37:22.938423 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:22.942790 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:22.941086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:49819)
0927 21:37:22.941819 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:22.942028 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":288,"threads_started":1}
W20260927 21:37:22.943400 20727 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:46229 (0 consecutive failures): Network error: Failed to ping master at 127.3.186.254:46229: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
I20260927 21:37:22.943930 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20599
I20260927 21:37:22.944190 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:22.943875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:47809)
0927 21:37:22.944013 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:22.944091 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:37:22.944623 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:44717
--local_ip_for_outbound_sockets=127.3.186.195
--tserver_master_addrs=127.3.186.254:46229
--webserver_port=40837
--webserver_interface=127.3.186.195
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:37:22.944970 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:22.944674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:33523)
0927 21:37:22.944802 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:22.944877 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:37:22.945200 20727 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:37:23.256222 20730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:23.256469 20730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:23.256625 20730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:23.263608 20730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:23.263720 20730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:23.263902 20730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:23.272701 20730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:44717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=40837
--tserver_master_addrs=127.3.186.254:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:23.273805 20730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:23.274957 20730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:23.287199 20735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:23.287266 20736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:23.287375 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:23.288148 20730 server_base.cc:1034] running on GCE node
I20260927 21:37:23.288695 20730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:23.289557 20730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:23.290777 20730 hybrid_clock.cc:648] HybridClock initialized: now 1790545043290726 us; error 43 us; skew 500 ppm
I20260927 21:37:23.291056 20730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:23 dist-test-slave-pqr8 krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545043, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:23.293979 20730 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:37:23.295226 20730 webserver.cc:460] Webserver started at http://127.3.186.195:40837/ using document root <none> and password file <none>
I20260927 21:37:23.295782 20730 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:23.295876 20730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:23.300565 20730 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:37:23.303011 20745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:23.304174 20730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:37:23.304293 20730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "687e89e535894539865ea65f0a276287"
format_stamp: "Formatted at 2026-09-27 21:36:37 on dist-test-slave-pqr8"
server_key: "8190892c134bd5e852afc21c03e4a45d"
server_key_iv: "e47296414f83403f8627e6ef86cda7cd"
server_key_version: "encryptionkey@0"
I20260927 21:37:23.305172 20730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:23.321915 20730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:23.325905 20730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:23.326159 20730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:23.327023 20730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:23.328190 20730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:23.328310 20730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:23.328445 20730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:23.328536 20730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:23.358930 20730 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:44717
I20260927 21:37:23.359014 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:44717 every 8 connection(s)
I20260927 21:37:23.360220 20730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:23.364336 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:23.362742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:44045)
0927 21:37:23.363401 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:23.363611 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":257,"threads_started":1}
W20260927 21:37:23.364900 20858 heartbeater.cc:644] Failed to heartbeat to 127.3.186.254:46229 (0 consecutive failures): Network error: Failed to ping master at 127.3.186.254:46229: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
I20260927 21:37:23.365465 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20730
I20260927 21:37:23.365706 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:23.365333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:49057)
0927 21:37:23.365472 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:23.365591 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:37:23.366587 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:23.366187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:49253)
0927 21:37:23.366347 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:23.366436 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 21:37:23.366856 20858 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:37:23.521076 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:23.520412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:54211)
0927 21:37:23.520796 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:23.520874 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":186,"threads_started":1}
I20260927 21:37:23.946426 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:23.945697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:45007)
0927 21:37:23.946117 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:23.946214 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260927 21:37:24.368009 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:24.367287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:57121)
0927 21:37:24.367716 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:24.367811 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260927 21:37:24.522512 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:24.521875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:33953)
0927 21:37:24.522298 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:24.522367 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":206,"threads_started":1}
I20260927 21:37:24.947877 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:24.947166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:37077)
0927 21:37:24.947578 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:24.947664 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":198,"threads_started":1}
I20260927 21:37:25.369472 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:25.368801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:44577)
0927 21:37:25.369238 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:25.369310 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260927 21:37:25.523908 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:25.523297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:45019)
0927 21:37:25.523686 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:25.523759 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":211,"threads_started":1}
I20260927 21:37:25.949280 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:25.948640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:45507)
0927 21:37:25.949029 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:25.949103 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":204,"threads_started":1}
I20260927 21:37:26.370894 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:26.370291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:40123)
0927 21:37:26.370671 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:26.370744 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":210,"threads_started":1}
I20260927 21:37:26.525238 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:26.524623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:37031)
0927 21:37:26.525004 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:26.525074 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":195,"threads_started":1}
I20260927 21:37:26.950764 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:26.950154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:36459)
0927 21:37:26.950525 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:26.950595 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":180,"threads_started":1}
I20260927 21:37:27.372402 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:27.371756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:59623)
0927 21:37:27.372149 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:27.372237 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":203,"threads_started":1}
I20260927 21:37:27.526659 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:27.525997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:36495)
0927 21:37:27.526439 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:27.526513 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":251,"threads_started":1}
I20260927 21:37:27.952289 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:27.951519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:52217)
0927 21:37:27.952012 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:27.952140 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":228,"threads_started":1}
I20260927 21:37:28.373857 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:28.373169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:51121)
0927 21:37:28.373627 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:28.373707 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":208,"threads_started":1}
I20260927 21:37:28.528023 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:28.527412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:41853)
0927 21:37:28.527806 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:28.527885 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":197,"threads_started":1}
I20260927 21:37:28.953874 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:28.953182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:40485)
0927 21:37:28.953643 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:28.953730 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260927 21:37:29.375223 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:29.374617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:53797)
0927 21:37:29.374997 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:29.375073 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":206,"threads_started":1}
I20260927 21:37:29.529779 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:29.528850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:40481)
0927 21:37:29.529376 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:29.529456 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":216,"threads_started":1}
I20260927 21:37:29.955431 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:29.954684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:47453)
0927 21:37:29.955148 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:29.955225 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":205,"threads_started":1}
I20260927 21:37:30.377002 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:30.376258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:39273)
0927 21:37:30.376727 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:30.376802 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":209,"threads_started":1}
I20260927 21:37:30.531242 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:30.530599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:46993)
0927 21:37:30.531017 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:30.531091 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":216,"threads_started":1}
I20260927 21:37:30.957088 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:30.956470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:60171)
0927 21:37:30.956874 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:30.956947 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":221,"threads_started":1}
I20260927 21:37:31.378499 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:31.377839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:55949)
0927 21:37:31.378289 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:31.378360 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
I20260927 21:37:31.532598 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:31.531960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:50615)
0927 21:37:31.532366 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:31.532460 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":208,"threads_started":1}
I20260927 21:37:31.958564 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:31.957916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:46043)
0927 21:37:31.958338 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:31.958410 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":199,"threads_started":1}
I20260927 21:37:32.379880 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:32.379242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:60079)
0927 21:37:32.379667 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:32.379737 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":207,"threads_started":1}
I20260927 21:37:32.534159 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:32.533409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:59213)
0927 21:37:32.533924 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:32.534012 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260927 21:37:32.959995 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:32.959318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:32963)
0927 21:37:32.959778 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:32.959850 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
I20260927 21:37:33.381335 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:33.380702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:58545)
0927 21:37:33.381121 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:33.381194 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260927 21:37:33.535595 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:33.534980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:37047)
0927 21:37:33.535365 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:33.535445 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":187,"threads_started":1}
I20260927 21:37:33.961570 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:33.960825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:50003)
0927 21:37:33.961255 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:33.961347 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260927 21:37:34.382786 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:34.382125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:44343)
0927 21:37:34.382537 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:34.382621 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":208,"threads_started":1}
I20260927 21:37:34.537043 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:34.536380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:38181)
0927 21:37:34.536811 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:34.536893 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260927 21:37:34.962989 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:34.962398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:60273)
0927 21:37:34.962781 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:34.962852 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":200,"threads_started":1}
I20260927 21:37:35.384238 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:35.383583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:46355)
0927 21:37:35.383987 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:35.384071 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":213,"threads_started":1}
I20260927 21:37:35.538460 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:35.537804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:55637)
0927 21:37:35.538222 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:35.538305 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":213,"threads_started":1}
I20260927 21:37:35.964433 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:35.963756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:54273)
0927 21:37:35.964184 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:35.964269 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":211,"threads_started":1}
I20260927 21:37:36.385715 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:36.385006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:49619)
0927 21:37:36.385428 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:36.385543 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260927 21:37:36.539775 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:36.539181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:33279)
0927 21:37:36.539551 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:36.539622 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":185,"threads_started":1}
I20260927 21:37:36.965819 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:36.965194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:52321)
0927 21:37:36.965594 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:36.965665 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":198,"threads_started":1}
I20260927 21:37:37.387261 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:37.386592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:35827)
0927 21:37:37.387015 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:37.387088 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":203,"threads_started":1}
I20260927 21:37:37.541489 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:37.540889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.193:60467)
0927 21:37:37.541261 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:37.541336 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":192,"threads_started":1}
I20260927 21:37:37.967537 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:37.966843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.194:50397)
0927 21:37:37.967286 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:37.967373 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260927 21:37:38.375424 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 20428
I20260927 21:37:38.389060 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:38.388294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46229 (local address 127.3.186.195:43977)
0927 21:37:38.388810 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:38.388895 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.254:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
I20260927 21:37:38.393208 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:38.376453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.3.186.193:33895)
0927 21:37:38.377061 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:38.377076 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:38.377118 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:37:38.378753 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:38.378763 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:38.379194 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:38.379495 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:38.380837 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:38.382130 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:38.383395 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:38.384126 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:38.384257 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:38.386917 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:38.386925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:38.386938 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:38.386977 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:38.390115 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:38.390751 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:38.390763 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:38.390769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:38.390919 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:38.391299 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:38.391304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:38.391307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:38.391779 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:38.392117 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:38.392588 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:37:38.392954 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
I20260927 21:37:38.394701 20591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:38.477316 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 20428
I20260927 21:37:38.494736 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 20599
I20260927 21:37:38.510877 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:38.495570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.3.186.194:46117)
0927 21:37:38.496029 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:38.496063 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:38.496099 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:37:38.497613 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:38.497622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:38.498020 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:38.498311 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:38.499542 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:38.500701 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:38.501862 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:38.502489 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:38.502611 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:38.504865 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:38.504871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:38.504882 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:38.504919 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:38.507892 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:38.508526 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:38.508532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:38.508539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:38.508685 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:38.509025 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:38.509028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:38.509030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:38.509467 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:38.509790 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:38.510332 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 21:37:38.510617 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":197,"threads_started":1}
I20260927 21:37:38.512117 20722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:38.593425 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 20599
I20260927 21:37:38.611311 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 20730
I20260927 21:37:38.628393 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:38.612277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.3.186.195:44717)
0927 21:37:38.612848 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:38.612862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:38.612902 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:37:38.614481 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:38.614489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:38.614901 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:38.615232 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:38.616607 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:38.617902 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:38.619210 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:38.619845 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:38.620047 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:38.622656 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:38.622663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:38.622677 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:38.622712 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:38.625297 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:38.626024 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:38.626045 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:38.626052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:38.626208 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:38.626581 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:38.626587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:38.626591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:38.627083 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:38.627395 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:38.627732 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 21:37:38.628137 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":249,"threads_started":1}
I20260927 21:37:38.629637 20853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:38.713122 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 20730
2026-09-27T21:37:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64202 ms)
[----------] 30 tests from SecurityITest (285354 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:37:38 dist-test-slave-pqr8 krb5kdc[20915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:37:38 dist-test-slave-pqr8 krb5kdc[20915](info): set up 2 sockets
Sep 27 21:37:38 dist-test-slave-pqr8 krb5kdc[20915](info): commencing operation
krb5kdc: starting...
W20260927 21:37:39.786448 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:37:39 dist-test-slave-pqr8 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:37:39.996723 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42511
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:40.310249 20928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:40.310488 20928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:40.310648 20928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:40.317474 20928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:40.317628 20928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:40.317732 20928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:40.317829 20928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:40.317907 20928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:40.327253 20928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:40.328385 20928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:40.329617 20928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:40.340941 20936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:40.341012 20933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:40.340987 20934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:40.341756 20928 server_base.cc:1034] running on GCE node
I20260927 21:37:40.342479 20928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:40.343750 20928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:40.344945 20928 hybrid_clock.cc:648] HybridClock initialized: now 1790545060344906 us; error 33 us; skew 500 ppm
I20260927 21:37:40.345220 20928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:40 dist-test-slave-pqr8 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:40.351637 20928 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:37:40.353171 20928 webserver.cc:460] Webserver started at http://127.3.186.254:44095/ using document root <none> and password file <none>
I20260927 21:37:40.353901 20928 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:40.354059 20928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:40.354393 20928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:40.357133 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "89e1970c4da04931b011c942039f66a3"
format_stamp: "Formatted at 2026-09-27 21:37:40 on dist-test-slave-pqr8"
server_key: "afd1e0435a501f410566eb59640d29f0"
server_key_iv: "90c3d06301bd96fe7f8557d304e4b397"
server_key_version: "encryptionkey@0"
I20260927 21:37:40.357955 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "89e1970c4da04931b011c942039f66a3"
format_stamp: "Formatted at 2026-09-27 21:37:40 on dist-test-slave-pqr8"
server_key: "afd1e0435a501f410566eb59640d29f0"
server_key_iv: "90c3d06301bd96fe7f8557d304e4b397"
server_key_version: "encryptionkey@0"
I20260927 21:37:40.362879 20928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:37:40.366034 20943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:40.367233 20928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 21:37:40.367410 20928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "89e1970c4da04931b011c942039f66a3"
format_stamp: "Formatted at 2026-09-27 21:37:40 on dist-test-slave-pqr8"
server_key: "afd1e0435a501f410566eb59640d29f0"
server_key_iv: "90c3d06301bd96fe7f8557d304e4b397"
server_key_version: "encryptionkey@0"
I20260927 21:37:40.367604 20928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:40.397734 20928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:40.401340 20928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:40.401669 20928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:40.419721 20928 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:42511
I20260927 21:37:40.419744 20994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:42511 every 8 connection(s)
I20260927 21:37:40.420912 20928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:40.424297 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 20928
I20260927 21:37:40.424506 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:37:40.424574 20995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:40.425271 3819 external_mini_cluster.cc:1428] Setting key 85fbca69707a356b2f4cc1734e2703da
I20260927 21:37:40.438499 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3: Bootstrap starting.
Sep 27 21:37:40 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:40.441836 20995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:40.443195 20995 log.cc:824] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:40.446123 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3: No bootstrap required, opened a new log
I20260927 21:37:40.448410 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:40.427237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.3.186.254:42511)
0927 21:37:40.428041 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:40.428051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:40.428088 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:37:40.429588 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:40.429611 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:40.429945 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:40.430261 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:40.431895 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:40.433029 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:40.434891 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:40.435523 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:40.435665 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:40.440358 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:40.440368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:40.440386 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:40.440430 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:40.444198 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:40.444883 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:40.444890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:40.444898 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:40.445093 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:40.445436 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:40.445441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:40.445444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:40.446117 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:40.446393 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:40.446779 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 21:37:40.447190 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":398,"threads_started":1}
I20260927 21:37:40.454312 20995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e1970c4da04931b011c942039f66a3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42511 } }
I20260927 21:37:40.454653 20995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:40.454789 20995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e1970c4da04931b011c942039f66a3, State: Initialized, Role: FOLLOWER
I20260927 21:37:40.455367 20995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e1970c4da04931b011c942039f66a3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42511 } }
I20260927 21:37:40.455623 20995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:40.455780 20995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:40.455937 20995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:40.458003 20995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e1970c4da04931b011c942039f66a3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42511 } }
I20260927 21:37:40.458470 20995 leader_election.cc:302] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e1970c4da04931b011c942039f66a3; no voters:
I20260927 21:37:40.459003 20995 leader_election.cc:288] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:37:40.459142 21000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:40.459784 21000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [term 1 LEADER]: Becoming Leader. State: Replica: 89e1970c4da04931b011c942039f66a3, State: Running, Role: LEADER
I20260927 21:37:40.460250 21000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e1970c4da04931b011c942039f66a3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42511 } }
I20260927 21:37:40.460906 20995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:40.463092 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e1970c4da04931b011c942039f66a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e1970c4da04931b011c942039f66a3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42511 } } }
I20260927 21:37:40.463056 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e1970c4da04931b011c942039f66a3. Latest consensus state: current_term: 1 leader_uuid: "89e1970c4da04931b011c942039f66a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e1970c4da04931b011c942039f66a3" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 42511 } } }
I20260927 21:37:40.463470 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:40.463470 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:40.465248 21007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:40.469427 21007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:40.477429 21007 catalog_manager.cc:1347] Generated new cluster ID: 2b57f1b0bef8418d9dc5d85aaa1c86a1
I20260927 21:37:40.477571 21007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:40.485391 21007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:37:40.486685 21007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:40.494338 21007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3: Generated new TSK 0
I20260927 21:37:40.495141 21007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:37:40.556792 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42511
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:40.868402 21021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:40.868674 21021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:40.868829 21021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:40.875726 21021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:40.875862 21021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:40.876031 21021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:40.884567 21021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:40.885617 21021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:40.886848 21021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:40.898914 21026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:40.898882 21029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:40.898962 21027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:40.899904 21021 server_base.cc:1034] running on GCE node
I20260927 21:37:40.900426 21021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:40.901203 21021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:40.902408 21021 hybrid_clock.cc:648] HybridClock initialized: now 1790545060902374 us; error 51 us; skew 500 ppm
I20260927 21:37:40.902693 21021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:40 dist-test-slave-pqr8 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:40.905632 21021 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:37:40.906802 21021 webserver.cc:460] Webserver started at http://127.3.186.193:45083/ using document root <none> and password file <none>
I20260927 21:37:40.907359 21021 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:40.907452 21021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:40.907697 21021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:40.910211 21021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "771d96a2d0b045e6a7676232db907f05"
format_stamp: "Formatted at 2026-09-27 21:37:40 on dist-test-slave-pqr8"
server_key: "e0580f5003bb8f8471d92aeaa2608374"
server_key_iv: "e9d8ea473e879c4644bce6b7f234311e"
server_key_version: "encryptionkey@0"
I20260927 21:37:40.910862 21021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "771d96a2d0b045e6a7676232db907f05"
format_stamp: "Formatted at 2026-09-27 21:37:40 on dist-test-slave-pqr8"
server_key: "e0580f5003bb8f8471d92aeaa2608374"
server_key_iv: "e9d8ea473e879c4644bce6b7f234311e"
server_key_version: "encryptionkey@0"
I20260927 21:37:40.915243 21021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:37:40.918241 21036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:40.919322 21021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:40.919427 21021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "771d96a2d0b045e6a7676232db907f05"
format_stamp: "Formatted at 2026-09-27 21:37:40 on dist-test-slave-pqr8"
server_key: "e0580f5003bb8f8471d92aeaa2608374"
server_key_iv: "e9d8ea473e879c4644bce6b7f234311e"
server_key_version: "encryptionkey@0"
I20260927 21:37:40.919567 21021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:40.946898 21021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:40.950874 21021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:40.951086 21021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:40.951831 21021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:40.953004 21021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:40.953078 21021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:40.953233 21021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:40.953310 21021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:40.984958 21021 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:39421
I20260927 21:37:40.985051 21148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:39421 every 8 connection(s)
I20260927 21:37:40.986217 21021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:40.987926 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21021
I20260927 21:37:40.988137 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:37:40.988946 3819 external_mini_cluster.cc:1428] Setting key ca72257a2991a5ae5bf300c0884aa95e
Sep 27 21:37:40 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:41.007880 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:40.988579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:58869 (local address 127.3.186.254:42511)
0927 21:37:40.989041 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:40.989048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:40.990010 (+ 962us) server_negotiation.cc:407] Connection header received
0927 21:37:40.992310 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:40.992313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:40.992370 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:40.992456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:40.994332 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:40.995111 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:40.997474 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:40.997911 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:40.998004 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:41.001532 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:41.001550 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:41.001557 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:41.001599 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:41.004879 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.005472 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.005480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.005486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.005653 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.006041 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.006048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.006054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.006425 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:41.006586 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:41.007158 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 21:37:41.007397 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260927 21:37:41.008265 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:40.989059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42511 (local address 127.3.186.193:58869)
0927 21:37:40.989830 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:40.989897 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:40.992051 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:40.992706 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:40.992719 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:40.993425 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:40.994155 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:40.994180 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:40.995261 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:40.995268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:40.997333 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:40.997351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:40.998131 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:40.998141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:40.998449 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:40.999067 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:40.999098 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:41.001310 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:41.005001 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:41.005012 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:41.005021 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:41.005351 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:41.005757 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:41.005759 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:41.005761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:41.005910 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:41.006542 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:41.006547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:41.006790 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 21:37:41.006998 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 21:37:41.007315 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260927 21:37:41.009685 21149 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42511
I20260927 21:37:41.009975 21149 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:41.010701 21149 heartbeater.cc:511] Master 127.3.186.254:42511 requested a full tablet report, sending...
I20260927 21:37:41.012949 20960 ts_manager.cc:194] Registered new tserver with Master: 771d96a2d0b045e6a7676232db907f05 (127.3.186.193:39421)
I20260927 21:37:41.014985 20960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:58869
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:37:41.041088 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42511
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:41.348598 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:41.348831 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:41.348913 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:41.355719 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:41.355803 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:41.355895 21155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:41.364708 21155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:41.365657 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:41.366765 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:41.378944 21163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:41.378983 21160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:41.379004 21161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:41.379729 21155 server_base.cc:1034] running on GCE node
I20260927 21:37:41.380239 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:41.381047 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:41.382264 21155 hybrid_clock.cc:648] HybridClock initialized: now 1790545061382226 us; error 48 us; skew 500 ppm
I20260927 21:37:41.382558 21155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:41 dist-test-slave-pqr8 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:41.385428 21155 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:37:41.386713 21155 webserver.cc:460] Webserver started at http://127.3.186.194:38633/ using document root <none> and password file <none>
I20260927 21:37:41.387334 21155 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:41.387485 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:41.387779 21155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:41.390501 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "850b253b676b41a3b01501a6a8283863"
format_stamp: "Formatted at 2026-09-27 21:37:41 on dist-test-slave-pqr8"
server_key: "6b6ddd8bf9e4c776e666ff7a103be3c7"
server_key_iv: "0d67f58337b4648434d10f86383a019d"
server_key_version: "encryptionkey@0"
I20260927 21:37:41.391216 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "850b253b676b41a3b01501a6a8283863"
format_stamp: "Formatted at 2026-09-27 21:37:41 on dist-test-slave-pqr8"
server_key: "6b6ddd8bf9e4c776e666ff7a103be3c7"
server_key_iv: "0d67f58337b4648434d10f86383a019d"
server_key_version: "encryptionkey@0"
I20260927 21:37:41.395723 21155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:37:41.398938 21170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:41.400194 21155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:37:41.400374 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "850b253b676b41a3b01501a6a8283863"
format_stamp: "Formatted at 2026-09-27 21:37:41 on dist-test-slave-pqr8"
server_key: "6b6ddd8bf9e4c776e666ff7a103be3c7"
server_key_iv: "0d67f58337b4648434d10f86383a019d"
server_key_version: "encryptionkey@0"
I20260927 21:37:41.400571 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:41.421837 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:41.425773 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:41.426062 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:41.426908 21155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:41.428196 21155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:41.428292 21155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:41.428431 21155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:41.428498 21155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:41.460984 21155 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:39787
I20260927 21:37:41.461097 21282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:39787 every 8 connection(s)
I20260927 21:37:41.462275 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:41.468856 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21155
I20260927 21:37:41.469029 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:37:41.469470 3819 external_mini_cluster.cc:1428] Setting key 4147f7a1d3ceed5ccc4cd5503a11c9ed
Sep 27 21:37:41 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:41.480832 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.464517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:32789 (local address 127.3.186.254:42511)
0927 21:37:41.464698 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:41.464702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:41.465802 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 21:37:41.467674 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:41.467679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:41.467782 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:41.467909 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:41.469363 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.470210 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.471841 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.472379 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.472503 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:41.475578 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:41.475583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:41.475586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:41.475615 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:41.478176 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.478839 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.478849 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.478852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.478965 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.479348 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.479353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.479356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.479608 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:41.479751 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:41.480299 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 21:37:41.480517 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:37:41.481665 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.464974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42511 (local address 127.3.186.194:32789)
0927 21:37:41.465671 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:41.465703 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:41.467409 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:41.468118 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:41.468126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:41.468613 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:41.469212 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:41.469229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.470347 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.470352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:41.471679 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:41.471691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.472600 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.472608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:41.472814 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:41.473418 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:41.473438 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:41.475377 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:41.478320 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:41.478329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:41.478341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:41.478715 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:41.479066 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:41.479072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:41.479077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:41.479241 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:41.479727 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:41.479733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:41.479993 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 21:37:41.480217 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 21:37:41.480659 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":254,"threads_started":1}
I20260927 21:37:41.482903 21283 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42511
I20260927 21:37:41.483168 21283 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:41.483778 21283 heartbeater.cc:511] Master 127.3.186.254:42511 requested a full tablet report, sending...
I20260927 21:37:41.485107 20960 ts_manager.cc:194] Registered new tserver with Master: 850b253b676b41a3b01501a6a8283863 (127.3.186.194:39787)
I20260927 21:37:41.486349 20960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:32789
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:37:41.517069 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42511
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:41.822014 21288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:41.822310 21288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:41.822467 21288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:41.829200 21288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:41.829327 21288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:41.829525 21288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:41.838310 21288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:42511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:41.839428 21288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:41.840631 21288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:41.852912 21293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:41.852877 21294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:41.852850 21296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:41.854050 21288 server_base.cc:1034] running on GCE node
I20260927 21:37:41.854590 21288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:41.855406 21288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:41.856619 21288 hybrid_clock.cc:648] HybridClock initialized: now 1790545061856579 us; error 56 us; skew 500 ppm
I20260927 21:37:41.856896 21288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:41 dist-test-slave-pqr8 krb5kdc[20915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:41.859766 21288 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:37:41.861065 21288 webserver.cc:460] Webserver started at http://127.3.186.195:39359/ using document root <none> and password file <none>
I20260927 21:37:41.861719 21288 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:41.861843 21288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:41.862180 21288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:41.864642 21288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "ab236d279dfc4add94ca559d7fcba1f8"
format_stamp: "Formatted at 2026-09-27 21:37:41 on dist-test-slave-pqr8"
server_key: "104d1044e05192a3cc9def5db2dfc755"
server_key_iv: "6031f0bd97fd0228f6595826bc845365"
server_key_version: "encryptionkey@0"
I20260927 21:37:41.865373 21288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "ab236d279dfc4add94ca559d7fcba1f8"
format_stamp: "Formatted at 2026-09-27 21:37:41 on dist-test-slave-pqr8"
server_key: "104d1044e05192a3cc9def5db2dfc755"
server_key_iv: "6031f0bd97fd0228f6595826bc845365"
server_key_version: "encryptionkey@0"
I20260927 21:37:41.870266 21288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:37:41.873330 21303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:41.874569 21288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:37:41.874729 21288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "ab236d279dfc4add94ca559d7fcba1f8"
format_stamp: "Formatted at 2026-09-27 21:37:41 on dist-test-slave-pqr8"
server_key: "104d1044e05192a3cc9def5db2dfc755"
server_key_iv: "6031f0bd97fd0228f6595826bc845365"
server_key_version: "encryptionkey@0"
I20260927 21:37:41.874907 21288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:41.895459 21288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:41.898912 21288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:41.899196 21288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:41.900017 21288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:41.901117 21288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:41.901216 21288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:41.901350 21288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:41.901417 21288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:41.933462 21288 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:32803
I20260927 21:37:41.933571 21415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:32803 every 8 connection(s)
I20260927 21:37:41.934598 21288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:41.944648 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21288
I20260927 21:37:41.944845 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:37:41.945295 3819 external_mini_cluster.cc:1428] Setting key 3a673a6eca7bb889e6b7c57798f5ed7f
Sep 27 21:37:41 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:41.952790 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.936768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:57387 (local address 127.3.186.254:42511)
0927 21:37:41.936975 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:41.936981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:41.937996 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 21:37:41.939740 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:41.939745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:41.939836 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:41.939944 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:41.941328 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.942098 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.943501 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.943877 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.944003 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:41.947643 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:41.947649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:41.947661 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:41.947691 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:41.950063 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.950728 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.950735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.950744 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.950859 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.951276 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.951282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.951285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.951529 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:41.951675 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:41.952257 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 21:37:41.952456 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:37:41.953567 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.937193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:42511 (local address 127.3.186.195:57387)
0927 21:37:41.937849 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:41.937883 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:41.939488 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:41.940170 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:41.940177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:41.940644 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:41.941145 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:41.941160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.942235 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.942241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:41.943361 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:41.943372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.944090 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.944098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:41.944343 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:41.945047 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:41.945068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:41.947453 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:41.950189 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:41.950196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:41.950209 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:41.950607 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:41.950967 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:41.950972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:41.950976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:41.951155 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:41.951647 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:41.951654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:41.951925 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 21:37:41.952168 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 21:37:41.952581 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":274,"threads_started":1}
I20260927 21:37:41.954766 21416 heartbeater.cc:347] Connected to a master server at 127.3.186.254:42511
I20260927 21:37:41.955005 21416 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:41.955605 21416 heartbeater.cc:511] Master 127.3.186.254:42511 requested a full tablet report, sending...
I20260927 21:37:41.956825 20960 ts_manager.cc:194] Registered new tserver with Master: ab236d279dfc4add94ca559d7fcba1f8 (127.3.186.195:32803)
I20260927 21:37:41.957656 20960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:57387
I20260927 21:37:41.960083 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:41.975632 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.964280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.3.186.254:42511)
0927 21:37:41.964412 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:41.964416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:41.964850 (+ 434us) server_negotiation.cc:407] Connection header received
0927 21:37:41.965014 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:41.965017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:41.965067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:41.965141 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:41.966659 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.967443 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.968606 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:41.968984 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:41.969081 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:41.971319 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:41.971324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:41.971326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:41.971350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:41.973606 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.974202 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.974205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.974207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.974313 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:41.974616 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:41.974618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:41.974620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:41.974873 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:41.975045 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:41.975320 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 21:37:41.975451 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:37:41.979070 20960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:41.985790 20960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:42.015586 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.997818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:32803 (local address 127.0.0.1:59482)
0927 21:37:42.007968 (+ 10150us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:42.007996 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:42.008149 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:42.008831 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:42.008838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:42.008921 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:42.009706 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.009719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.011225 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.011232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.013158 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.013168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.014601 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.014607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.014785 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.014836 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:37:42.014944 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:37:42.015117 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10002,"mutex_wait_us":335,"spinlock_wait_cycles":52352,"thread_start_us":9850,"threads_started":1}
I20260927 21:37:42.016191 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.999141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.3.186.195:32803)
0927 21:37:42.007729 (+ 8588us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.007738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.008280 (+ 542us) server_negotiation.cc:407] Connection header received
0927 21:37:42.008374 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.008382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.008597 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.009205 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:42.009912 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.011088 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.013292 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.014419 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.014535 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.015634 (+ 1099us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.015703 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.015853 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8426,"thread_start_us":8273,"threads_started":1}
I20260927 21:37:42.016364 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.997142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:39421 (local address 127.0.0.1:49488)
0927 21:37:41.997823 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:41.997871 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:41.998018 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:42.008831 (+ 10813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:42.008838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:42.008907 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:42.009302 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.009316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.011523 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.011528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.013795 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.013815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.015555 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.015564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.015727 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.015755 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:37:42.015816 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:37:42.015921 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":73,"spinlock_wait_cycles":52224,"thread_start_us":319,"threads_started":1}
I20260927 21:37:42.016535 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.998203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:39787 (local address 127.0.0.1:49368)
0927 21:37:42.008766 (+ 10563us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:42.008789 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:42.009021 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:42.009641 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:42.009645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:42.009671 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:42.010063 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.010078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.011878 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.011883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.013951 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.013969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.015729 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.015734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.015885 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.015921 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:37:42.016007 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:37:42.016127 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10434,"spinlock_wait_cycles":211072,"thread_start_us":10327,"threads_started":1}
I20260927 21:37:42.016748 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.998690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.3.186.194:39787)
0927 21:37:42.007248 (+ 8558us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.007259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.008943 (+ 1684us) server_negotiation.cc:407] Connection header received
0927 21:37:42.009232 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.009239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.009440 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.009656 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:42.010233 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.011711 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.014090 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.015260 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.015391 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.015585 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.016215 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.016498 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8410,"thread_start_us":8321,"threads_started":1}
I20260927 21:37:42.017658 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:41.997344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.3.186.193:39421)
0927 21:37:42.008326 (+ 10982us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.008334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.008355 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:37:42.008424 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.008429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.008597 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.009552 (+ 955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:42.009780 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.011366 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.013967 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.015302 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.016977 (+ 1675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.017196 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.017264 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.017365 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10848,"thread_start_us":10719,"threads_started":1}
I20260927 21:37:42.017720 21149 heartbeater.cc:503] Master 127.3.186.254:42511 was elected leader, sending a full tablet report...
I20260927 21:37:42.019892 21218 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca5bd0e48864c1aabafa787f8aeaac8 (DEFAULT_TABLE table=test-table [id=fe0994eea3e6458eaf7adf5b6184d45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:42.019892 21351 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca5bd0e48864c1aabafa787f8aeaac8 (DEFAULT_TABLE table=test-table [id=fe0994eea3e6458eaf7adf5b6184d45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:42.021266 21218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca5bd0e48864c1aabafa787f8aeaac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:42.021267 21351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca5bd0e48864c1aabafa787f8aeaac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:42.021562 21084 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca5bd0e48864c1aabafa787f8aeaac8 (DEFAULT_TABLE table=test-table [id=fe0994eea3e6458eaf7adf5b6184d45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:42.022764 21084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca5bd0e48864c1aabafa787f8aeaac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:42.033107 21434 tablet_bootstrap.cc:492] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Bootstrap starting.
I20260927 21:37:42.034909 21435 tablet_bootstrap.cc:492] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Bootstrap starting.
I20260927 21:37:42.035617 21436 tablet_bootstrap.cc:492] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Bootstrap starting.
I20260927 21:37:42.036545 21434 tablet_bootstrap.cc:654] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:42.037810 21434 log.cc:824] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:42.038391 21435 tablet_bootstrap.cc:654] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:42.038894 21436 tablet_bootstrap.cc:654] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:42.039608 21435 log.cc:824] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:42.039921 21436 log.cc:824] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:42.041096 21434 tablet_bootstrap.cc:492] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: No bootstrap required, opened a new log
I20260927 21:37:42.041339 21434 ts_tablet_manager.cc:1397] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:37:42.042409 21436 tablet_bootstrap.cc:492] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: No bootstrap required, opened a new log
I20260927 21:37:42.042448 21435 tablet_bootstrap.cc:492] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: No bootstrap required, opened a new log
I20260927 21:37:42.042655 21436 ts_tablet_manager.cc:1397] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 21:37:42.042673 21435 ts_tablet_manager.cc:1397] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 21:37:42.050757 21435 raft_consensus.cc:348] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.050721 21436 raft_consensus.cc:348] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.051081 21435 raft_consensus.cc:374] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:42.051103 21436 raft_consensus.cc:374] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:42.051194 21436 raft_consensus.cc:729] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850b253b676b41a3b01501a6a8283863, State: Initialized, Role: FOLLOWER
I20260927 21:37:42.051198 21435 raft_consensus.cc:729] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab236d279dfc4add94ca559d7fcba1f8, State: Initialized, Role: FOLLOWER
I20260927 21:37:42.051754 21435 consensus_queue.cc:260] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.051744 21436 consensus_queue.cc:260] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.052378 21434 raft_consensus.cc:348] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.052739 21416 heartbeater.cc:503] Master 127.3.186.254:42511 was elected leader, sending a full tablet report...
I20260927 21:37:42.052887 21434 raft_consensus.cc:374] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:42.053010 21434 raft_consensus.cc:729] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d96a2d0b045e6a7676232db907f05, State: Initialized, Role: FOLLOWER
I20260927 21:37:42.053256 21283 heartbeater.cc:503] Master 127.3.186.254:42511 was elected leader, sending a full tablet report...
I20260927 21:37:42.053570 21434 consensus_queue.cc:260] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.053913 21436 ts_tablet_manager.cc:1428] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 21:37:42.055778 21434 ts_tablet_manager.cc:1428] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260927 21:37:42.056625 21435 ts_tablet_manager.cc:1428] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Time spent starting tablet: real 0.014s user 0.008s sys 0.003s
I20260927 21:37:42.101976 21442 raft_consensus.cc:482] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:42.102164 21442 raft_consensus.cc:504] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.103753 21442 leader_election.cc:288] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 850b253b676b41a3b01501a6a8283863 (127.3.186.194:39787), ab236d279dfc4add94ca559d7fcba1f8 (127.3.186.195:32803)
I20260927 21:37:42.112516 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.104577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:46355 (local address 127.3.186.195:32803)
0927 21:37:42.104766 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.104774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.105875 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 21:37:42.106093 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.106098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.106164 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.106232 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:42.106861 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.107898 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.110238 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.111459 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.111623 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.111692 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.112155 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.112277 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:37:42.112517 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.104029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:59729 (local address 127.3.186.194:39787)
0927 21:37:42.104252 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.104258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.105537 (+ 1279us) server_negotiation.cc:407] Connection header received
0927 21:37:42.105809 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.105813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.105872 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.105942 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:42.106581 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.107619 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.110235 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.111455 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.111639 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.111699 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.112155 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.112277 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:37:42.112776 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.103897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:39787 (local address 127.3.186.193:59729)
0927 21:37:42.105407 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:42.105429 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:42.105663 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:42.105976 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:42.105983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:42.106033 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:42.106429 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.106442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.107747 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.107750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.110000 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.110024 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.111762 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.111767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.111971 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.112031 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:37:42.112116 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:37:42.112299 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"mutex_wait_us":78,"thread_start_us":1617,"threads_started":1}
I20260927 21:37:42.112802 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.104583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:32803 (local address 127.3.186.193:46355)
0927 21:37:42.105778 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:42.105795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:42.105921 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:42.106282 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:42.106286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:42.106316 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:42.106724 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.106732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.108017 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.108022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.110074 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:42.110086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.111740 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.111751 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:42.111977 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.112020 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:37:42.112116 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:37:42.112299 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":119,"thread_start_us":471,"threads_started":1}
I20260927 21:37:42.113749 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca5bd0e48864c1aabafa787f8aeaac8" candidate_uuid: "771d96a2d0b045e6a7676232db907f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850b253b676b41a3b01501a6a8283863" is_pre_election: true
I20260927 21:37:42.113732 21371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca5bd0e48864c1aabafa787f8aeaac8" candidate_uuid: "771d96a2d0b045e6a7676232db907f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab236d279dfc4add94ca559d7fcba1f8" is_pre_election: true
I20260927 21:37:42.114259 21238 raft_consensus.cc:2454] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771d96a2d0b045e6a7676232db907f05 in term 0.
I20260927 21:37:42.114257 21371 raft_consensus.cc:2454] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 771d96a2d0b045e6a7676232db907f05 in term 0.
I20260927 21:37:42.114872 21039 leader_election.cc:302] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771d96a2d0b045e6a7676232db907f05, ab236d279dfc4add94ca559d7fcba1f8; no voters:
I20260927 21:37:42.115264 21442 raft_consensus.cc:2781] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:42.115422 21442 raft_consensus.cc:482] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:42.115538 21442 raft_consensus.cc:3037] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:42.117977 21442 raft_consensus.cc:504] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.118459 21442 leader_election.cc:288] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [CANDIDATE]: Term 1 election: Requested vote from peers 850b253b676b41a3b01501a6a8283863 (127.3.186.194:39787), ab236d279dfc4add94ca559d7fcba1f8 (127.3.186.195:32803)
I20260927 21:37:42.118871 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca5bd0e48864c1aabafa787f8aeaac8" candidate_uuid: "771d96a2d0b045e6a7676232db907f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850b253b676b41a3b01501a6a8283863"
I20260927 21:37:42.119058 21238 raft_consensus.cc:3037] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:42.121124 21238 raft_consensus.cc:2454] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771d96a2d0b045e6a7676232db907f05 in term 1.
I20260927 21:37:42.121495 21037 leader_election.cc:302] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 771d96a2d0b045e6a7676232db907f05, 850b253b676b41a3b01501a6a8283863; no voters:
I20260927 21:37:42.121505 21371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca5bd0e48864c1aabafa787f8aeaac8" candidate_uuid: "771d96a2d0b045e6a7676232db907f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab236d279dfc4add94ca559d7fcba1f8"
I20260927 21:37:42.121754 21371 raft_consensus.cc:3037] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:42.121810 21442 raft_consensus.cc:2781] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:42.122257 21442 raft_consensus.cc:686] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 1 LEADER]: Becoming Leader. State: Replica: 771d96a2d0b045e6a7676232db907f05, State: Running, Role: LEADER
I20260927 21:37:42.122829 21442 consensus_queue.cc:237] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } }
I20260927 21:37:42.124832 21371 raft_consensus.cc:2454] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 771d96a2d0b045e6a7676232db907f05 in term 1.
I20260927 21:37:42.127442 20959 catalog_manager.cc:5495] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 771d96a2d0b045e6a7676232db907f05 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "771d96a2d0b045e6a7676232db907f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771d96a2d0b045e6a7676232db907f05" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 39421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 } health_report { overall_health: UNKNOWN } } }
I20260927 21:37:42.169926 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.164574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.3.186.193:39421)
0927 21:37:42.164773 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.164778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.164906 (+ 128us) server_negotiation.cc:407] Connection header received
0927 21:37:42.165158 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.165162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.165236 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.165370 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:42.165739 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.166496 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.167973 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.168376 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.168505 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.168966 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:42.169384 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:42.169561 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.169625 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.169696 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:37:42.176694 21371 raft_consensus.cc:1261] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 1 FOLLOWER]: Refusing update from remote peer 771d96a2d0b045e6a7676232db907f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:42.176787 21238 raft_consensus.cc:1261] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 1 FOLLOWER]: Refusing update from remote peer 771d96a2d0b045e6a7676232db907f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:42.177646 21445 consensus_queue.cc:1035] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab236d279dfc4add94ca559d7fcba1f8" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:42.178011 21442 consensus_queue.cc:1035] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "850b253b676b41a3b01501a6a8283863" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:37:42.186080 21417 tablet.cc:2369] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:42.192348 21450 mvcc.cc:205] Tried to move back new op lower bound from 7334072574668013568 to 7334072574459191296. Current Snapshot: MvccSnapshot[applied={T|T < 7334072574668013568}]
I20260927 21:37:42.195578 21449 mvcc.cc:205] Tried to move back new op lower bound from 7334072574668013568 to 7334072574459191296. Current Snapshot: MvccSnapshot[applied={T|T < 7334072574668013568}]
I20260927 21:37:42.209681 20959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:42.209935 20959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:42.212971 20959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89e1970c4da04931b011c942039f66a3: Sending DeleteTablet for 3 replicas of tablet 5ca5bd0e48864c1aabafa787f8aeaac8
I20260927 21:37:42.213986 21084 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca5bd0e48864c1aabafa787f8aeaac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:42 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:49488
I20260927 21:37:42.214144 21351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca5bd0e48864c1aabafa787f8aeaac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:42 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:59482
I20260927 21:37:42.214833 21459 tablet_replica.cc:330] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: stopping tablet replica
I20260927 21:37:42.215269 21459 raft_consensus.cc:2229] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:42.215780 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21021
I20260927 21:37:42.215826 21218 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca5bd0e48864c1aabafa787f8aeaac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:42 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:49368
I20260927 21:37:42.215962 21459 raft_consensus.cc:2258] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:42.217867 21459 ts_tablet_manager.cc:1905] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:42.226984 21459 ts_tablet_manager.cc:1918] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:42.227193 21459 log.cc:1196] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/5ca5bd0e48864c1aabafa787f8aeaac8
I20260927 21:37:42.227744 21459 ts_tablet_manager.cc:1939] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 771d96a2d0b045e6a7676232db907f05: Deleting consensus metadata
I20260927 21:37:42.229334 20944 catalog_manager.cc:4841] TS 771d96a2d0b045e6a7676232db907f05 (127.3.186.193:39421): tablet 5ca5bd0e48864c1aabafa787f8aeaac8 (table test-table [id=fe0994eea3e6458eaf7adf5b6184d45c]) successfully deleted
I20260927 21:37:42.230146 21461 tablet_replica.cc:330] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: stopping tablet replica
I20260927 21:37:42.230590 21461 raft_consensus.cc:2229] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:42.231071 21461 raft_consensus.cc:2258] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:42.232993 21461 ts_tablet_manager.cc:1905] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:42.233923 21460 tablet_replica.cc:330] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: stopping tablet replica
I20260927 21:37:42.234371 21460 raft_consensus.cc:2229] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:42.234892 21460 raft_consensus.cc:2258] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:42.236733 21460 ts_tablet_manager.cc:1905] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:37:42 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:37:42.239735 21461 ts_tablet_manager.cc:1918] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:42.239953 21461 log.cc:1196] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/5ca5bd0e48864c1aabafa787f8aeaac8
I20260927 21:37:42.240562 21461 ts_tablet_manager.cc:1939] T 5ca5bd0e48864c1aabafa787f8aeaac8 P 850b253b676b41a3b01501a6a8283863: Deleting consensus metadata
I20260927 21:37:42.242197 20944 catalog_manager.cc:4841] TS 850b253b676b41a3b01501a6a8283863 (127.3.186.194:39787): tablet 5ca5bd0e48864c1aabafa787f8aeaac8 (table test-table [id=fe0994eea3e6458eaf7adf5b6184d45c]) successfully deleted
I20260927 21:37:42.243438 21460 ts_tablet_manager.cc:1918] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:42.243580 21460 log.cc:1196] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/5ca5bd0e48864c1aabafa787f8aeaac8
I20260927 21:37:42.244130 21460 ts_tablet_manager.cc:1939] T 5ca5bd0e48864c1aabafa787f8aeaac8 P ab236d279dfc4add94ca559d7fcba1f8: Deleting consensus metadata
I20260927 21:37:42.245682 20946 catalog_manager.cc:4841] TS ab236d279dfc4add94ca559d7fcba1f8 (127.3.186.195:32803): tablet 5ca5bd0e48864c1aabafa787f8aeaac8 (table test-table [id=fe0994eea3e6458eaf7adf5b6184d45c]) successfully deleted
I20260927 21:37:42.245824 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.217959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.3.186.193:39421)
0927 21:37:42.218383 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.218391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.226951 (+ 8560us) server_negotiation.cc:407] Connection header received
0927 21:37:42.227427 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.227434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.227513 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.227602 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:42.229735 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.230946 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.232931 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.233695 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.233787 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.239234 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:42.239247 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:42.239260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:42.239300 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:42.242997 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:42.243662 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:42.243671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:42.243676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:42.243829 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:42.244215 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:42.244221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:42.244224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:42.244529 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:42.244841 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.245073 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.245296 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260927 21:37:42.246482 21144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:42.340965 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21021
I20260927 21:37:42.359519 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21155
Sep 27 21:37:42 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:37:42.373951 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.360239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.3.186.194:39787)
0927 21:37:42.360395 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.360399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.360416 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:37:42.360538 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.360541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.360602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.360684 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:42.362048 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.362788 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.364000 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.364402 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.364497 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.368683 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:42.368694 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:42.368708 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:42.368741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:42.371567 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:42.372223 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:42.372229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:42.372232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:42.372346 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:42.372663 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:42.372667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:42.372670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:42.373053 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:42.373264 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.373433 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.373635 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:37:42.374562 21278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:42.461802 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21155
I20260927 21:37:42.478677 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21288
Sep 27 21:37:42 dist-test-slave-pqr8 krb5kdc[20915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:37:42.493402 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:42.479490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.3.186.195:32803)
0927 21:37:42.479673 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:42.479678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:42.479697 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:37:42.479861 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:42.479865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:42.479928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:42.480010 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:42.481209 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.481978 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.483224 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:42.483600 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:42.483684 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:42.487995 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:42.488009 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:42.488024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:42.488061 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:42.490791 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:42.491461 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:42.491468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:42.491470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:42.491610 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:42.491968 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:42.491974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:42.491977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:42.492305 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:42.492618 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:42.492922 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:37:42.493107 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:37:42.494062 21411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:42.578990 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21288
I20260927 21:37:42.595016 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 20928
I20260927 21:37:42.596074 20990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:42.701643 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 20928
2026-09-27T21:37:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3990 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T21:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:37:42Z Disabled control of system clock
I20260927 21:37:42.771589 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:37531
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:37531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:37:43.076478 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:43.076728 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:43.076804 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:43.083614 21470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:43.083698 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:43.083740 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:43.083792 21470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:43.083833 21470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:43.092420 21470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:37531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:37531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:43.093410 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:43.095119 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:43.106141 21475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:43.106087 21478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:43.106117 21476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:43.107550 21470 server_base.cc:1034] running on GCE node
I20260927 21:37:43.108244 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:43.109375 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:43.110562 21470 hybrid_clock.cc:648] HybridClock initialized: now 1790545063110524 us; error 66 us; skew 500 ppm
I20260927 21:37:43.110869 21470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:43.112802 21470 webserver.cc:460] Webserver started at http://127.3.186.254:34791/ using document root <none> and password file <none>
I20260927 21:37:43.113400 21470 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:43.113504 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:43.113809 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:43.116385 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "3238d1448397481e9938f077bb1ede93"
format_stamp: "Formatted at 2026-09-27 21:37:43 on dist-test-slave-pqr8"
server_key: "6fa5b682b68ef34cda2c9dee7567c599"
server_key_iv: "3c045e51ac6cae1f06a698ae60a34cc3"
server_key_version: "encryptionkey@0"
I20260927 21:37:43.117149 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "3238d1448397481e9938f077bb1ede93"
format_stamp: "Formatted at 2026-09-27 21:37:43 on dist-test-slave-pqr8"
server_key: "6fa5b682b68ef34cda2c9dee7567c599"
server_key_iv: "3c045e51ac6cae1f06a698ae60a34cc3"
server_key_version: "encryptionkey@0"
I20260927 21:37:43.122905 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:37:43.125926 21484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:43.127138 21470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260927 21:37:43.127310 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "3238d1448397481e9938f077bb1ede93"
format_stamp: "Formatted at 2026-09-27 21:37:43 on dist-test-slave-pqr8"
server_key: "6fa5b682b68ef34cda2c9dee7567c599"
server_key_iv: "3c045e51ac6cae1f06a698ae60a34cc3"
server_key_version: "encryptionkey@0"
I20260927 21:37:43.127506 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:43.156985 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:43.157781 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:43.158051 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:43.175024 21470 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:37531
I20260927 21:37:43.175060 21535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:37531 every 8 connection(s)
I20260927 21:37:43.176216 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:43.178661 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21470
I20260927 21:37:43.178833 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:37:43.179363 3819 external_mini_cluster.cc:1428] Setting key 458f9ca89ca4d966f006b7c45f4defb3
I20260927 21:37:43.180011 21536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:43.194191 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93: Bootstrap starting.
I20260927 21:37:43.194334 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:43.181414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.3.186.254:37531)
0927 21:37:43.182591 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:43.182600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:43.182641 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:37:43.184257 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:43.184276 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:43.184677 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:43.184979 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:43.185606 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:43.186879 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:43.188827 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:43.189564 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:43.189740 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:43.191374 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:43.191388 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:43.191407 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:43.192165 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:43.192266 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:43.192301 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:43.192417 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:43.192682 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 21:37:43.193017 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":495,"threads_started":1}
I20260927 21:37:43.197968 21536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:43.199167 21536 log.cc:824] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:43.201891 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93: No bootstrap required, opened a new log
I20260927 21:37:43.208927 21536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3238d1448397481e9938f077bb1ede93" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37531 } }
I20260927 21:37:43.209270 21536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:43.209372 21536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3238d1448397481e9938f077bb1ede93, State: Initialized, Role: FOLLOWER
I20260927 21:37:43.209957 21536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3238d1448397481e9938f077bb1ede93" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37531 } }
I20260927 21:37:43.210191 21536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:43.210327 21536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:43.210493 21536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:43.212783 21536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3238d1448397481e9938f077bb1ede93" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37531 } }
I20260927 21:37:43.213241 21536 leader_election.cc:302] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3238d1448397481e9938f077bb1ede93; no voters:
I20260927 21:37:43.213815 21536 leader_election.cc:288] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:37:43.213894 21541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:43.214587 21541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [term 1 LEADER]: Becoming Leader. State: Replica: 3238d1448397481e9938f077bb1ede93, State: Running, Role: LEADER
I20260927 21:37:43.215070 21541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3238d1448397481e9938f077bb1ede93" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37531 } }
I20260927 21:37:43.215646 21536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:43.218014 21542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3238d1448397481e9938f077bb1ede93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3238d1448397481e9938f077bb1ede93" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37531 } } }
I20260927 21:37:43.218492 21542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:43.217635 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3238d1448397481e9938f077bb1ede93. Latest consensus state: current_term: 1 leader_uuid: "3238d1448397481e9938f077bb1ede93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3238d1448397481e9938f077bb1ede93" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 37531 } } }
I20260927 21:37:43.219161 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:43.221686 21548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:43.227531 21548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:43.236398 21548 catalog_manager.cc:1347] Generated new cluster ID: 50354c6219d44626b3a41b31a7630b29
I20260927 21:37:43.236487 21548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:43.275507 21548 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:37:43.276676 21548 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:43.284390 21548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93: Generated new TSK 0
I20260927 21:37:43.285061 21548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:37:43.298568 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:37531
--builtin_ntp_servers=127.3.186.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:37:43.614941 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:43.615183 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:43.615262 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:43.622287 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:43.622370 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:43.622558 21560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:43.631536 21560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:37531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:43.632632 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:43.634336 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:43.647028 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:43.647004 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:43.647089 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:43.650837 21560 server_base.cc:1034] running on GCE node
I20260927 21:37:43.651361 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:43.652117 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:43.653301 21560 hybrid_clock.cc:648] HybridClock initialized: now 1790545063653264 us; error 26 us; skew 500 ppm
I20260927 21:37:43.653644 21560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:43.655625 21560 webserver.cc:460] Webserver started at http://127.3.186.193:44235/ using document root <none> and password file <none>
I20260927 21:37:43.656222 21560 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:43.656303 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:43.656591 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:43.659192 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "89cdf91e85b84e58a98246da08a98597"
format_stamp: "Formatted at 2026-09-27 21:37:43 on dist-test-slave-pqr8"
server_key: "67762d9c70f4b24874d0d20ebd7a2221"
server_key_iv: "26deaee50e0dc01b37cb12e12ced5099"
server_key_version: "encryptionkey@0"
I20260927 21:37:43.659900 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "89cdf91e85b84e58a98246da08a98597"
format_stamp: "Formatted at 2026-09-27 21:37:43 on dist-test-slave-pqr8"
server_key: "67762d9c70f4b24874d0d20ebd7a2221"
server_key_iv: "26deaee50e0dc01b37cb12e12ced5099"
server_key_version: "encryptionkey@0"
I20260927 21:37:43.664449 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:37:43.667649 21574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:43.668798 21560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:37:43.668958 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "89cdf91e85b84e58a98246da08a98597"
format_stamp: "Formatted at 2026-09-27 21:37:43 on dist-test-slave-pqr8"
server_key: "67762d9c70f4b24874d0d20ebd7a2221"
server_key_iv: "26deaee50e0dc01b37cb12e12ced5099"
server_key_version: "encryptionkey@0"
I20260927 21:37:43.669159 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:43.695680 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:43.696364 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:43.696643 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:43.697625 21560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:43.698902 21560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:43.699018 21560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:43.699146 21560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:43.699231 21560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:43.730374 21560 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:34447
I20260927 21:37:43.730535 21686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:34447 every 8 connection(s)
I20260927 21:37:43.731539 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:43.732290 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21560
I20260927 21:37:43.732445 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:37:43.733006 3819 external_mini_cluster.cc:1428] Setting key 4d5c07b65ade98625efaf8249750080b
I20260927 21:37:43.736016 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:37531
--builtin_ntp_servers=127.3.186.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:37:43.746122 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:43.734094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:56783 (local address 127.3.186.254:37531)
0927 21:37:43.734501 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:43.734505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:43.735841 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 21:37:43.738630 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:43.738634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:43.738689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:43.738772 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:43.740050 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:43.740767 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:43.743136 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:43.743750 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:43.743863 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:43.745255 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:43.745260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:43.745263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:43.745278 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:43.745312 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:43.745323 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:43.745417 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:43.745678 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:37:43.745767 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":235,"threads_started":1}
I20260927 21:37:43.747009 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:43.734664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:37531 (local address 127.3.186.193:56783)
0927 21:37:43.735691 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:43.735736 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:43.738353 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:43.739014 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:43.739026 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:43.739132 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:43.739870 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:43.739896 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:43.740909 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:43.740917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:43.742983 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:43.743001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:43.743959 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:43.743968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:43.744280 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:43.744969 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:43.744971 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:43.745011 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:43.745075 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:43.745416 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:43.745427 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:43.745436 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:37:43.745673 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 21:37:43.746015 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":311,"threads_started":1}
I20260927 21:37:43.748503 21687 heartbeater.cc:347] Connected to a master server at 127.3.186.254:37531
I20260927 21:37:43.748771 21687 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:43.749388 21687 heartbeater.cc:511] Master 127.3.186.254:37531 requested a full tablet report, sending...
I20260927 21:37:43.751399 21501 ts_manager.cc:194] Registered new tserver with Master: 89cdf91e85b84e58a98246da08a98597 (127.3.186.193:34447)
I20260927 21:37:43.753376 21501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.193:56783
W20260927 21:37:44.047688 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:44.047909 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:44.047987 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:44.054613 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:44.054694 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:44.054785 21691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:44.063181 21691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:37531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:44.064118 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:44.065606 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:44.077533 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:44.077592 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:44.077569 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:44.078653 21691 server_base.cc:1034] running on GCE node
I20260927 21:37:44.079167 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:44.079962 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:44.081177 21691 hybrid_clock.cc:648] HybridClock initialized: now 1790545064081149 us; error 43 us; skew 500 ppm
I20260927 21:37:44.081468 21691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:44.083474 21691 webserver.cc:460] Webserver started at http://127.3.186.194:34711/ using document root <none> and password file <none>
I20260927 21:37:44.084101 21691 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:44.084223 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:44.084520 21691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:44.087076 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "6091b1815a7f4d88859cda5085e6c950"
format_stamp: "Formatted at 2026-09-27 21:37:44 on dist-test-slave-pqr8"
server_key: "eb37d2d062f3610093a2a4c35c6a4b67"
server_key_iv: "0c817c142a939815bd590df0c726acfc"
server_key_version: "encryptionkey@0"
I20260927 21:37:44.087819 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "6091b1815a7f4d88859cda5085e6c950"
format_stamp: "Formatted at 2026-09-27 21:37:44 on dist-test-slave-pqr8"
server_key: "eb37d2d062f3610093a2a4c35c6a4b67"
server_key_iv: "0c817c142a939815bd590df0c726acfc"
server_key_version: "encryptionkey@0"
I20260927 21:37:44.092444 21691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 21:37:44.095647 21705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:44.096829 21691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:37:44.096987 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "6091b1815a7f4d88859cda5085e6c950"
format_stamp: "Formatted at 2026-09-27 21:37:44 on dist-test-slave-pqr8"
server_key: "eb37d2d062f3610093a2a4c35c6a4b67"
server_key_iv: "0c817c142a939815bd590df0c726acfc"
server_key_version: "encryptionkey@0"
I20260927 21:37:44.097222 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:44.119942 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:44.120609 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:44.120885 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:44.121966 21691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:44.123574 21691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:44.123682 21691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:44.123770 21691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:44.123844 21691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:44.155077 21691 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:40261
I20260927 21:37:44.155190 21817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:40261 every 8 connection(s)
I20260927 21:37:44.156246 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:44.161561 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21691
I20260927 21:37:44.161717 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:37:44.162263 3819 external_mini_cluster.cc:1428] Setting key c11df8fa48d94b2ab9888ee97640614d
I20260927 21:37:44.165221 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:37531
--builtin_ntp_servers=127.3.186.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:37:44.169155 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.158503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:56321 (local address 127.3.186.254:37531)
0927 21:37:44.158674 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.158678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.159774 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 21:37:44.161615 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.161621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.161699 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.161827 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:44.163080 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.164091 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.166435 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.166800 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.166965 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.168463 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:44.168468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:44.168471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:44.168486 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:44.168513 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:44.168524 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:44.168635 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.168874 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.168944 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:37:44.170615 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.158866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:37531 (local address 127.3.186.194:56321)
0927 21:37:44.159635 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.159673 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.161257 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.162072 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.162082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.162194 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:44.162894 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.162912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.164235 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.164240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.166298 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.166315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.167102 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.167111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.167422 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.168126 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:44.168132 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:44.168177 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:44.168282 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:44.168642 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:44.168653 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:44.168663 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.168904 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.169283 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":232,"threads_started":1}
I20260927 21:37:44.171908 21818 heartbeater.cc:347] Connected to a master server at 127.3.186.254:37531
I20260927 21:37:44.172228 21818 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:44.172864 21818 heartbeater.cc:511] Master 127.3.186.254:37531 requested a full tablet report, sending...
I20260927 21:37:44.174371 21501 ts_manager.cc:194] Registered new tserver with Master: 6091b1815a7f4d88859cda5085e6c950 (127.3.186.194:40261)
I20260927 21:37:44.175438 21501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.194:56321
W20260927 21:37:44.471909 21821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:44.472172 21821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:44.472316 21821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:44.479323 21821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:44.479440 21821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:44.479609 21821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:44.487946 21821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:37531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:44.488983 21821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:44.490622 21821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:44.502878 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:44.503039 21829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:44.503396 21821 server_base.cc:1034] running on GCE node
W20260927 21:37:44.503610 21826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:44.504204 21821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:44.505082 21821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:44.506304 21821 hybrid_clock.cc:648] HybridClock initialized: now 1790545064506262 us; error 26 us; skew 500 ppm
I20260927 21:37:44.506609 21821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:44.508695 21821 webserver.cc:460] Webserver started at http://127.3.186.195:37517/ using document root <none> and password file <none>
I20260927 21:37:44.509317 21821 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:44.509441 21821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:44.509766 21821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:44.512253 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "dffa50d6e1f74d33b7950b312421a1b7"
format_stamp: "Formatted at 2026-09-27 21:37:44 on dist-test-slave-pqr8"
server_key: "3e9efd328705f9bb56a03d9cb321b4a2"
server_key_iv: "4932d81d48048835a52e6f740865fe80"
server_key_version: "encryptionkey@0"
I20260927 21:37:44.512970 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "dffa50d6e1f74d33b7950b312421a1b7"
format_stamp: "Formatted at 2026-09-27 21:37:44 on dist-test-slave-pqr8"
server_key: "3e9efd328705f9bb56a03d9cb321b4a2"
server_key_iv: "4932d81d48048835a52e6f740865fe80"
server_key_version: "encryptionkey@0"
I20260927 21:37:44.517582 21821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 21:37:44.520645 21835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:44.521780 21821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:44.521927 21821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "dffa50d6e1f74d33b7950b312421a1b7"
format_stamp: "Formatted at 2026-09-27 21:37:44 on dist-test-slave-pqr8"
server_key: "3e9efd328705f9bb56a03d9cb321b4a2"
server_key_iv: "4932d81d48048835a52e6f740865fe80"
server_key_version: "encryptionkey@0"
I20260927 21:37:44.522145 21821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:44.532913 21821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:44.533664 21821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:44.533941 21821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:44.534752 21821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:44.535988 21821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:44.536105 21821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:44.536232 21821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:44.536317 21821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:44.567438 21821 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:37323
I20260927 21:37:44.567514 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:37323 every 8 connection(s)
I20260927 21:37:44.568576 21821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:44.569344 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 21821
I20260927 21:37:44.569526 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:37:44.570053 3819 external_mini_cluster.cc:1428] Setting key 14b4d718ad2fd3917c8a17b6990b9e88
I20260927 21:37:44.580962 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.571176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:53461 (local address 127.3.186.254:37531)
0927 21:37:44.571369 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.571375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.573748 (+ 2373us) server_negotiation.cc:407] Connection header received
0927 21:37:44.575296 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.575300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.575370 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.575441 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:44.576326 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.577008 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.578576 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.578981 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.579117 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.580335 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:44.580340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:44.580343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:44.580359 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:44.580390 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:44.580401 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:44.580475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.580659 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.580731 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:37:44.581844 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.571700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:37531 (local address 127.3.186.195:53461)
0927 21:37:44.573594 (+ 1894us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.573646 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.575094 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.575613 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.575619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.575710 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:44.576199 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.576215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.577170 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.577173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.578414 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.578425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.579210 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.579216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.579393 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.580029 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:44.580032 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:44.580094 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:44.580157 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:44.580486 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:44.580494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:44.580498 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.580625 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.580857 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":1238,"threads_started":1}
I20260927 21:37:44.583072 21948 heartbeater.cc:347] Connected to a master server at 127.3.186.254:37531
I20260927 21:37:44.583348 21948 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:44.583936 21948 heartbeater.cc:511] Master 127.3.186.254:37531 requested a full tablet report, sending...
I20260927 21:37:44.585273 21500 ts_manager.cc:194] Registered new tserver with Master: dffa50d6e1f74d33b7950b312421a1b7 (127.3.186.195:37323)
I20260927 21:37:44.586216 21500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.195:53461
I20260927 21:37:44.597165 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:44.610975 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.605623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.3.186.254:37531)
0927 21:37:44.605802 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.605806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.606289 (+ 483us) server_negotiation.cc:407] Connection header received
0927 21:37:44.606412 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.606415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.606479 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.606557 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:44.606987 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.607645 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.609104 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.609480 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.609628 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.610479 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:44.610483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:44.610486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:44.610501 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:44.610527 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:44.610538 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:44.610610 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.610717 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.610778 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:37:44.614233 21500 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:44.620903 21500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:44.653693 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.631917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:40261 (local address 127.0.0.1:45548)
0927 21:37:44.643190 (+ 11273us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.643231 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.643402 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.644618 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.644629 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.644668 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:44.645164 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.645180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.647767 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.647775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.650641 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.650657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.652834 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.652840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.653067 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.653111 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.653220 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.653371 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10986,"thread_start_us":13246,"threads_started":1}
I20260927 21:37:44.654466 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.632206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.3.186.194:40261)
0927 21:37:44.643936 (+ 11730us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.643947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.643979 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:37:44.644093 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.644103 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.644380 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.644586 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:44.645433 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.647530 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.650847 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.652108 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.652442 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.652717 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.654077 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.654211 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11553,"thread_start_us":12631,"threads_started":1}
I20260927 21:37:44.657493 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.633013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.3.186.193:34447)
0927 21:37:44.649912 (+ 16899us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.649920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.649949 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:37:44.650066 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.650073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.650281 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.650476 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:44.651050 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.652373 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.654840 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.656213 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.656351 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.656570 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.657030 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.657205 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16752,"thread_start_us":17225,"threads_started":1}
I20260927 21:37:44.657665 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.632129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:37323 (local address 127.0.0.1:42338)
0927 21:37:44.645495 (+ 13366us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.645531 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.645650 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.651504 (+ 5854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.651509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.651533 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:44.651832 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.651841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.653581 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.653586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.655295 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.655305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.656858 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.656865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.657013 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.657057 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.657179 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.657295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13219,"mutex_wait_us":1,"thread_start_us":10704,"threads_started":1}
I20260927 21:37:44.657745 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.632777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.3.186.195:37323)
0927 21:37:44.650989 (+ 18212us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.650997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.651035 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:37:44.651123 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.651130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.651317 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.651482 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:44.652028 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.653446 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.655457 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.656684 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.656815 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.657101 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.657218 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.657356 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18050,"thread_start_us":18220,"threads_started":1}
I20260927 21:37:44.658172 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.632482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:34447 (local address 127.0.0.1:60336)
0927 21:37:44.649616 (+ 17134us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.649633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.649742 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.650482 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.650487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.650509 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:44.650911 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.650919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.652527 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.652532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.654598 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.654612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.656695 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.656701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.656842 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.656870 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.657753 (+ 883us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.657866 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16997,"mutex_wait_us":42,"thread_start_us":16826,"threads_started":1}
I20260927 21:37:44.658174 21753 tablet_service.cc:1467] Processing CreateTablet for tablet 0a22624139f94d8db819427d0ca8c09a (DEFAULT_TABLE table=test-table [id=d1edc516c82f456ebdaf41b0a6ea793b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:44.659482 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a22624139f94d8db819427d0ca8c09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:44.660936 21883 tablet_service.cc:1467] Processing CreateTablet for tablet 0a22624139f94d8db819427d0ca8c09a (DEFAULT_TABLE table=test-table [id=d1edc516c82f456ebdaf41b0a6ea793b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:44.661343 21622 tablet_service.cc:1467] Processing CreateTablet for tablet 0a22624139f94d8db819427d0ca8c09a (DEFAULT_TABLE table=test-table [id=d1edc516c82f456ebdaf41b0a6ea793b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:44.662569 21883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a22624139f94d8db819427d0ca8c09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:44.662583 21622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a22624139f94d8db819427d0ca8c09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:44.670734 21966 tablet_bootstrap.cc:492] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Bootstrap starting.
I20260927 21:37:44.674301 21966 tablet_bootstrap.cc:654] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:44.675217 21966 log.cc:824] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:44.677735 21966 tablet_bootstrap.cc:492] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: No bootstrap required, opened a new log
I20260927 21:37:44.677835 21967 tablet_bootstrap.cc:492] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Bootstrap starting.
I20260927 21:37:44.678023 21966 ts_tablet_manager.cc:1397] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 21:37:44.681928 21967 tablet_bootstrap.cc:654] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:44.682304 21969 tablet_bootstrap.cc:492] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Bootstrap starting.
I20260927 21:37:44.683040 21967 log.cc:824] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:44.685266 21967 tablet_bootstrap.cc:492] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: No bootstrap required, opened a new log
I20260927 21:37:44.685506 21969 tablet_bootstrap.cc:654] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:44.685541 21967 ts_tablet_manager.cc:1397] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:37:44.686647 21969 log.cc:824] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:44.687086 21966 raft_consensus.cc:348] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.687520 21966 raft_consensus.cc:374] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:44.687619 21966 raft_consensus.cc:729] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6091b1815a7f4d88859cda5085e6c950, State: Initialized, Role: FOLLOWER
I20260927 21:37:44.688297 21966 consensus_queue.cc:260] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.689363 21818 heartbeater.cc:503] Master 127.3.186.254:37531 was elected leader, sending a full tablet report...
I20260927 21:37:44.689415 21969 tablet_bootstrap.cc:492] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: No bootstrap required, opened a new log
I20260927 21:37:44.689670 21969 ts_tablet_manager.cc:1397] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 21:37:44.690086 21966 ts_tablet_manager.cc:1428] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260927 21:37:44.695112 21967 raft_consensus.cc:348] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.695484 21967 raft_consensus.cc:374] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:44.695583 21967 raft_consensus.cc:729] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cdf91e85b84e58a98246da08a98597, State: Initialized, Role: FOLLOWER
I20260927 21:37:44.696112 21967 consensus_queue.cc:260] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.697428 21687 heartbeater.cc:503] Master 127.3.186.254:37531 was elected leader, sending a full tablet report...
I20260927 21:37:44.697710 21967 ts_tablet_manager.cc:1428] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 21:37:44.699637 21969 raft_consensus.cc:348] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.699956 21969 raft_consensus.cc:374] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:44.700047 21969 raft_consensus.cc:729] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffa50d6e1f74d33b7950b312421a1b7, State: Initialized, Role: FOLLOWER
I20260927 21:37:44.700637 21969 consensus_queue.cc:260] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.701711 21948 heartbeater.cc:503] Master 127.3.186.254:37531 was elected leader, sending a full tablet report...
I20260927 21:37:44.702186 21969 ts_tablet_manager.cc:1428] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260927 21:37:44.733763 21688 tablet.cc:2369] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:44.741303 21973 raft_consensus.cc:482] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:44.741556 21973 raft_consensus.cc:504] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.743041 21973 leader_election.cc:288] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6091b1815a7f4d88859cda5085e6c950 (127.3.186.194:40261), dffa50d6e1f74d33b7950b312421a1b7 (127.3.186.195:37323)
I20260927 21:37:44.749634 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.743388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:33305 (local address 127.3.186.194:40261)
0927 21:37:44.743598 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.743605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.744332 (+ 727us) server_negotiation.cc:407] Connection header received
0927 21:37:44.744529 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.744538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.744592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.744686 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:44.745199 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.746119 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.747994 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.748758 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.748933 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.749001 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.749282 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.749393 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:37:44.749697 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.743600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:39329 (local address 127.3.186.195:37323)
0927 21:37:44.743767 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.743774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.744444 (+ 670us) server_negotiation.cc:407] Connection header received
0927 21:37:44.744627 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.744631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.744679 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.744769 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:44.745335 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.746298 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.748060 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.748822 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.748995 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.749058 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.749354 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.749466 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:37:44.749795 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.743521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:37323 (local address 127.3.186.193:39329)
0927 21:37:44.744335 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.744348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.744489 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.744788 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.744792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.744826 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:44.745177 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.745185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.746427 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.746433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.747884 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.747895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.749012 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.749018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.749204 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.749225 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.749319 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.749428 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":42,"thread_start_us":502,"threads_started":1}
I20260927 21:37:44.750043 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.743461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:40261 (local address 127.3.186.193:33305)
0927 21:37:44.744226 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:44.744241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:44.744393 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:44.744698 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:44.744703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:44.744724 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:44.745055 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.745062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.746246 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.746254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.747802 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:44.747814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.748939 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.748945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:44.749118 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.749153 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:37:44.749257 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:37:44.749370 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":21,"thread_start_us":442,"threads_started":1}
I20260927 21:37:44.750756 21903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a22624139f94d8db819427d0ca8c09a" candidate_uuid: "89cdf91e85b84e58a98246da08a98597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffa50d6e1f74d33b7950b312421a1b7" is_pre_election: true
I20260927 21:37:44.750973 21773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a22624139f94d8db819427d0ca8c09a" candidate_uuid: "89cdf91e85b84e58a98246da08a98597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6091b1815a7f4d88859cda5085e6c950" is_pre_election: true
I20260927 21:37:44.751161 21903 raft_consensus.cc:2454] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cdf91e85b84e58a98246da08a98597 in term 0.
I20260927 21:37:44.751469 21773 raft_consensus.cc:2454] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cdf91e85b84e58a98246da08a98597 in term 0.
I20260927 21:37:44.751911 21577 leader_election.cc:302] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cdf91e85b84e58a98246da08a98597, dffa50d6e1f74d33b7950b312421a1b7; no voters:
I20260927 21:37:44.752409 21973 raft_consensus.cc:2781] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:44.752560 21973 raft_consensus.cc:482] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:44.752671 21973 raft_consensus.cc:3037] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:44.754714 21973 raft_consensus.cc:504] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.755234 21973 leader_election.cc:288] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [CANDIDATE]: Term 1 election: Requested vote from peers 6091b1815a7f4d88859cda5085e6c950 (127.3.186.194:40261), dffa50d6e1f74d33b7950b312421a1b7 (127.3.186.195:37323)
I20260927 21:37:44.755606 21773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a22624139f94d8db819427d0ca8c09a" candidate_uuid: "89cdf91e85b84e58a98246da08a98597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6091b1815a7f4d88859cda5085e6c950"
I20260927 21:37:44.755635 21903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a22624139f94d8db819427d0ca8c09a" candidate_uuid: "89cdf91e85b84e58a98246da08a98597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffa50d6e1f74d33b7950b312421a1b7"
I20260927 21:37:44.755797 21903 raft_consensus.cc:3037] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:44.755797 21773 raft_consensus.cc:3037] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:44.758935 21903 raft_consensus.cc:2454] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cdf91e85b84e58a98246da08a98597 in term 1.
I20260927 21:37:44.758935 21773 raft_consensus.cc:2454] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cdf91e85b84e58a98246da08a98597 in term 1.
I20260927 21:37:44.759311 21577 leader_election.cc:302] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cdf91e85b84e58a98246da08a98597, dffa50d6e1f74d33b7950b312421a1b7; no voters:
I20260927 21:37:44.759595 21973 raft_consensus.cc:2781] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:44.759938 21973 raft_consensus.cc:686] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 1 LEADER]: Becoming Leader. State: Replica: 89cdf91e85b84e58a98246da08a98597, State: Running, Role: LEADER
I20260927 21:37:44.760324 21973 consensus_queue.cc:237] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } }
I20260927 21:37:44.764298 21500 catalog_manager.cc:5495] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89cdf91e85b84e58a98246da08a98597 (127.3.186.193). New cstate: current_term: 1 leader_uuid: "89cdf91e85b84e58a98246da08a98597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cdf91e85b84e58a98246da08a98597" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 34447 } health_report { overall_health: HEALTHY } } }
I20260927 21:37:44.779771 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.775235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.3.186.193:34447)
0927 21:37:44.775391 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.775395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.775451 (+ 56us) server_negotiation.cc:407] Connection header received
0927 21:37:44.775596 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.775600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.775679 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.775780 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:44.776063 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.776702 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.778113 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.778493 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.778587 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.778955 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:44.779336 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:44.779437 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.779536 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.779597 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:37:44.786654 21903 raft_consensus.cc:1261] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 1 FOLLOWER]: Refusing update from remote peer 89cdf91e85b84e58a98246da08a98597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:44.786653 21773 raft_consensus.cc:1261] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 1 FOLLOWER]: Refusing update from remote peer 89cdf91e85b84e58a98246da08a98597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:44.787477 21977 consensus_queue.cc:1035] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6091b1815a7f4d88859cda5085e6c950" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:44.787834 21973 consensus_queue.cc:1035] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dffa50d6e1f74d33b7950b312421a1b7" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:44.799309 21982 mvcc.cc:205] Tried to move back new op lower bound from 7334072585357975552 to 7334072585261629440. Current Snapshot: MvccSnapshot[applied={T|T < 7334072585357975552}]
I20260927 21:37:44.809268 21500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:44.809502 21500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:44.812755 21500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3238d1448397481e9938f077bb1ede93: Sending DeleteTablet for 3 replicas of tablet 0a22624139f94d8db819427d0ca8c09a
I20260927 21:37:44.813724 21883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a22624139f94d8db819427d0ca8c09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:44 UTC) from {username='slave'} at 127.0.0.1:42338
I20260927 21:37:44.813724 21753 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a22624139f94d8db819427d0ca8c09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:44 UTC) from {username='slave'} at 127.0.0.1:45548
I20260927 21:37:44.813918 21622 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a22624139f94d8db819427d0ca8c09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:44 UTC) from {username='slave'} at 127.0.0.1:60336
I20260927 21:37:44.821803 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21560
I20260927 21:37:44.826650 21991 tablet_replica.cc:330] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: stopping tablet replica
I20260927 21:37:44.827225 21991 raft_consensus.cc:2229] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:44.827346 21992 tablet_replica.cc:330] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: stopping tablet replica
I20260927 21:37:44.827695 21992 raft_consensus.cc:2229] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:44.827881 21991 raft_consensus.cc:2258] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:44.828250 21992 raft_consensus.cc:2258] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:44.828855 21990 tablet_replica.cc:330] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: stopping tablet replica
I20260927 21:37:44.830093 21992 ts_tablet_manager.cc:1905] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:44.830170 21991 ts_tablet_manager.cc:1905] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:44.831873 21990 raft_consensus.cc:2229] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:44.832440 21990 raft_consensus.cc:2258] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:44.837015 21992 ts_tablet_manager.cc:1918] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:44.837172 21992 log.cc:1196] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/0a22624139f94d8db819427d0ca8c09a
I20260927 21:37:44.837720 21992 ts_tablet_manager.cc:1939] T 0a22624139f94d8db819427d0ca8c09a P 89cdf91e85b84e58a98246da08a98597: Deleting consensus metadata
I20260927 21:37:44.838850 21990 ts_tablet_manager.cc:1905] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:44.839287 21485 catalog_manager.cc:4841] TS 89cdf91e85b84e58a98246da08a98597 (127.3.186.193:34447): tablet 0a22624139f94d8db819427d0ca8c09a (table test-table [id=d1edc516c82f456ebdaf41b0a6ea793b]) successfully deleted
I20260927 21:37:44.840718 21991 ts_tablet_manager.cc:1918] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:44.840888 21991 log.cc:1196] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/0a22624139f94d8db819427d0ca8c09a
I20260927 21:37:44.841363 21991 ts_tablet_manager.cc:1939] T 0a22624139f94d8db819427d0ca8c09a P dffa50d6e1f74d33b7950b312421a1b7: Deleting consensus metadata
I20260927 21:37:44.841552 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.823019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.3.186.193:34447)
0927 21:37:44.823388 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.823395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.833578 (+ 10183us) server_negotiation.cc:407] Connection header received
0927 21:37:44.833743 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.833748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.833834 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.833953 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:44.834391 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.835591 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.837413 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.838107 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.838221 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.839968 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:44.839982 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:44.839995 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:44.840651 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:44.840699 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:44.840719 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:44.840806 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.841060 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.841173 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260927 21:37:44.842474 21682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:44.842971 21487 catalog_manager.cc:4841] TS dffa50d6e1f74d33b7950b312421a1b7 (127.3.186.195:37323): tablet 0a22624139f94d8db819427d0ca8c09a (table test-table [id=d1edc516c82f456ebdaf41b0a6ea793b]) successfully deleted
I20260927 21:37:44.846563 21990 ts_tablet_manager.cc:1918] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:44.846731 21990 log.cc:1196] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/0a22624139f94d8db819427d0ca8c09a
I20260927 21:37:44.847277 21990 ts_tablet_manager.cc:1939] T 0a22624139f94d8db819427d0ca8c09a P 6091b1815a7f4d88859cda5085e6c950: Deleting consensus metadata
I20260927 21:37:44.848645 21488 catalog_manager.cc:4841] TS 6091b1815a7f4d88859cda5085e6c950 (127.3.186.194:40261): tablet 0a22624139f94d8db819427d0ca8c09a (table test-table [id=d1edc516c82f456ebdaf41b0a6ea793b]) successfully deleted
I20260927 21:37:44.934798 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21560
I20260927 21:37:44.951761 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21691
I20260927 21:37:44.958508 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:44.952457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.3.186.194:40261)
0927 21:37:44.952624 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:44.952631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:44.952687 (+ 56us) server_negotiation.cc:407] Connection header received
0927 21:37:44.952902 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:44.952906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:44.953000 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:44.953128 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:44.953523 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.954215 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.955531 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:44.955907 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:44.956043 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:44.957093 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:44.957102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:44.957115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:44.957738 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:44.957784 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:44.957809 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:44.957923 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:44.958088 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:37:44.958189 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:37:44.959110 21813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:45.043154 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21691
I20260927 21:37:45.059690 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21821
I20260927 21:37:45.066624 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:45.060443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.3.186.195:37323)
0927 21:37:45.060635 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:45.060642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:45.060667 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:37:45.060884 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:45.060894 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:45.060977 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:45.061089 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:45.061477 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:45.062241 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:45.063488 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:45.063889 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:45.064017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:45.065085 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:45.065094 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:45.065108 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:45.065882 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:45.065928 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:45.065963 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:45.066092 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:45.066223 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:37:45.066326 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:37:45.067229 21943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:45.153404 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21821
I20260927 21:37:45.169902 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 21470
I20260927 21:37:45.170850 21531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:45.286581 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 21470
2026-09-27T21:37:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2583 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:37:45 dist-test-slave-pqr8 krb5kdc[22000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:37:45 dist-test-slave-pqr8 krb5kdc[22000](info): set up 2 sockets
Sep 27 21:37:45 dist-test-slave-pqr8 krb5kdc[22000](info): commencing operation
krb5kdc: starting...
W20260927 21:37:46.361539 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:37:46 dist-test-slave-pqr8 krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:37:46.564464 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44287
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:46.864025 22013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:46.864315 22013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:46.864485 22013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:46.871069 22013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:46.871199 22013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:46.871297 22013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:46.871394 22013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:46.871475 22013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:46.880364 22013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:44287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:46.881440 22013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:46.882551 22013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:46.893440 22018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:46.893538 22019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:46.893489 22021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:46.894395 22013 server_base.cc:1034] running on GCE node
I20260927 21:37:46.895028 22013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:46.896217 22013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:46.897423 22013 hybrid_clock.cc:648] HybridClock initialized: now 1790545066897389 us; error 42 us; skew 500 ppm
I20260927 21:37:46.897747 22013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:46 dist-test-slave-pqr8 krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545066, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:46.900482 22013 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:37:46.901875 22013 webserver.cc:460] Webserver started at http://127.3.186.254:39885/ using document root <none> and password file <none>
I20260927 21:37:46.902462 22013 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:46.902599 22013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:46.902876 22013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:46.905656 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "99d038e5c69b458b9eb19926a848d7ac"
format_stamp: "Formatted at 2026-09-27 21:37:46 on dist-test-slave-pqr8"
server_key: "55c45972eddf15cc09ea2f26ff0c71bf"
server_key_iv: "07ea937c584edb582e03c883f0f5daa8"
server_key_version: "encryptionkey@0"
I20260927 21:37:46.906356 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "99d038e5c69b458b9eb19926a848d7ac"
format_stamp: "Formatted at 2026-09-27 21:37:46 on dist-test-slave-pqr8"
server_key: "55c45972eddf15cc09ea2f26ff0c71bf"
server_key_iv: "07ea937c584edb582e03c883f0f5daa8"
server_key_version: "encryptionkey@0"
I20260927 21:37:46.910822 22013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:37:46.913791 22028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:46.914877 22013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:46.915048 22013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "99d038e5c69b458b9eb19926a848d7ac"
format_stamp: "Formatted at 2026-09-27 21:37:46 on dist-test-slave-pqr8"
server_key: "55c45972eddf15cc09ea2f26ff0c71bf"
server_key_iv: "07ea937c584edb582e03c883f0f5daa8"
server_key_version: "encryptionkey@0"
I20260927 21:37:46.915239 22013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:46.927330 22013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:46.930830 22013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:46.931084 22013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:46.948510 22013 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:44287
I20260927 21:37:46.948581 22079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:44287 every 8 connection(s)
I20260927 21:37:46.949682 22013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:46.950369 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22013
I20260927 21:37:46.950562 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:37:46.951076 3819 external_mini_cluster.cc:1428] Setting key 7fee7358c7f53fe623c0050cd5265b95
I20260927 21:37:46.953714 22080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:37:46 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:46.965131 22080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac: Bootstrap starting.
I20260927 21:37:46.968838 22080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:46.969974 22080 log.cc:824] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:46.972937 22080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac: No bootstrap required, opened a new log
I20260927 21:37:46.973954 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:46.952711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.3.186.254:44287)
0927 21:37:46.953465 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:46.953478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:46.953556 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:37:46.955681 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:46.955705 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:46.956156 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:46.956549 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:46.957803 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:46.959763 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:46.961203 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:46.962182 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:46.962353 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:46.966165 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:46.966174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:46.966189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:46.966236 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:46.969930 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:46.970692 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:46.970697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:46.970702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:46.970846 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:46.971291 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:46.971298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:46.971304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:46.971864 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:46.972145 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:46.972440 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 21:37:46.972787 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":320,"threads_started":1}
I20260927 21:37:46.980433 22080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d038e5c69b458b9eb19926a848d7ac" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44287 } }
I20260927 21:37:46.980712 22080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:46.980788 22080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d038e5c69b458b9eb19926a848d7ac, State: Initialized, Role: FOLLOWER
I20260927 21:37:46.981216 22080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d038e5c69b458b9eb19926a848d7ac" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44287 } }
I20260927 21:37:46.981389 22080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:46.981477 22080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:46.981673 22080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:46.983762 22080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d038e5c69b458b9eb19926a848d7ac" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44287 } }
I20260927 21:37:46.984126 22080 leader_election.cc:302] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99d038e5c69b458b9eb19926a848d7ac; no voters:
I20260927 21:37:46.984638 22080 leader_election.cc:288] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:37:46.984751 22085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:46.985603 22085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [term 1 LEADER]: Becoming Leader. State: Replica: 99d038e5c69b458b9eb19926a848d7ac, State: Running, Role: LEADER
I20260927 21:37:46.986109 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d038e5c69b458b9eb19926a848d7ac" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44287 } }
I20260927 21:37:46.986459 22080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:46.988920 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99d038e5c69b458b9eb19926a848d7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d038e5c69b458b9eb19926a848d7ac" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44287 } } }
I20260927 21:37:46.988950 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99d038e5c69b458b9eb19926a848d7ac. Latest consensus state: current_term: 1 leader_uuid: "99d038e5c69b458b9eb19926a848d7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d038e5c69b458b9eb19926a848d7ac" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 44287 } } }
I20260927 21:37:46.989702 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:46.989789 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:46.994752 22090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:46.998839 22090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:47.006520 22090 catalog_manager.cc:1347] Generated new cluster ID: f3d2890d6797482f93f81de95846cab7
I20260927 21:37:47.006608 22090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:47.010473 22102 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:37:47.014353 22090 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:37:47.015398 22090 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:47.027679 22090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac: Generated new TSK 0
I20260927 21:37:47.028503 22090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:37:47.090754 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44287
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:47.398867 22106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:47.399107 22106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:47.399191 22106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:47.406075 22106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:47.406160 22106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:47.406265 22106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:47.415158 22106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:47.416111 22106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:47.417203 22106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:47.429934 22112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:47.429993 22111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:47.430037 22114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:47.430794 22106 server_base.cc:1034] running on GCE node
I20260927 21:37:47.431346 22106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:47.432164 22106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:47.433394 22106 hybrid_clock.cc:648] HybridClock initialized: now 1790545067433344 us; error 60 us; skew 500 ppm
I20260927 21:37:47.433782 22106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:47 dist-test-slave-pqr8 krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:47.436795 22106 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:37:47.438220 22106 webserver.cc:460] Webserver started at http://127.3.186.193:37573/ using document root <none> and password file <none>
I20260927 21:37:47.438827 22106 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:47.438966 22106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:47.439248 22106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:47.441808 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "3e52b198de624442bf185a8d5764062a"
format_stamp: "Formatted at 2026-09-27 21:37:47 on dist-test-slave-pqr8"
server_key: "ed67a3a6d5cc7882640d812283373650"
server_key_iv: "bfd0b1538b79543de1fb168546af040b"
server_key_version: "encryptionkey@0"
I20260927 21:37:47.442541 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "3e52b198de624442bf185a8d5764062a"
format_stamp: "Formatted at 2026-09-27 21:37:47 on dist-test-slave-pqr8"
server_key: "ed67a3a6d5cc7882640d812283373650"
server_key_iv: "bfd0b1538b79543de1fb168546af040b"
server_key_version: "encryptionkey@0"
I20260927 21:37:47.447082 22106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:37:47.450196 22121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:47.451282 22106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:47.451442 22106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "3e52b198de624442bf185a8d5764062a"
format_stamp: "Formatted at 2026-09-27 21:37:47 on dist-test-slave-pqr8"
server_key: "ed67a3a6d5cc7882640d812283373650"
server_key_iv: "bfd0b1538b79543de1fb168546af040b"
server_key_version: "encryptionkey@0"
I20260927 21:37:47.451619 22106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:47.483270 22106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:47.487193 22106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:47.487498 22106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:47.488348 22106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:47.489686 22106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:47.489779 22106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:47.489928 22106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:47.490011 22106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:47.520411 22106 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:33765
I20260927 21:37:47.520480 22233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:33765 every 8 connection(s)
I20260927 21:37:47.521505 22106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:47.528704 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22106
I20260927 21:37:47.528867 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:37:47.529386 3819 external_mini_cluster.cc:1428] Setting key c74d898cffe652a84e27ab08a91d1c7a
Sep 27 21:37:47 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:47.540184 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:47.523747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:56775 (local address 127.3.186.254:44287)
0927 21:37:47.524158 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:47.524164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:47.525087 (+ 923us) server_negotiation.cc:407] Connection header received
0927 21:37:47.526892 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:47.526896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:47.526958 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:47.527091 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:47.528418 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:47.529487 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:47.531251 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:47.531624 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:47.531759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:47.534962 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:47.534967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:47.534970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:47.535003 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:47.537432 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:47.538108 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:47.538116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:47.538121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:47.538239 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:47.538658 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:47.538663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:47.538668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:47.538896 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:47.539050 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:47.539640 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 21:37:47.539835 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":190,"threads_started":1}
I20260927 21:37:47.540917 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:47.524286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44287 (local address 127.3.186.193:56775)
0927 21:37:47.524914 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:47.524949 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:47.526633 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:47.527273 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:47.527280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:47.527738 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:47.528232 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:47.528249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:47.529666 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:47.529673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:47.531117 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:47.531129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:47.531853 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:47.531860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:47.532060 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:47.532669 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:47.532691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:47.534768 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:47.537587 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:47.537595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:47.537608 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:47.537990 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:47.538359 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:47.538365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:47.538368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:47.538544 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:47.539016 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:47.539023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:47.539301 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 21:37:47.539541 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 21:37:47.539963 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":241,"threads_started":1}
I20260927 21:37:47.542399 22234 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44287
I20260927 21:37:47.542680 22234 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:47.543278 22234 heartbeater.cc:511] Master 127.3.186.254:44287 requested a full tablet report, sending...
I20260927 21:37:47.545204 22045 ts_manager.cc:194] Registered new tserver with Master: 3e52b198de624442bf185a8d5764062a (127.3.186.193:33765)
I20260927 21:37:47.547348 22045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:56775
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:37:47.580179 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44287
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:47.891064 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:47.891294 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:47.891381 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:47.898322 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:47.898419 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:47.898520 22240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:47.907537 22240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:47.908552 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:47.909719 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:47.922008 22245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:47.922076 22248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:47.922165 22246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:47.922676 22240 server_base.cc:1034] running on GCE node
I20260927 21:37:47.923208 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:47.924021 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:47.925241 22240 hybrid_clock.cc:648] HybridClock initialized: now 1790545067925199 us; error 56 us; skew 500 ppm
I20260927 21:37:47.925576 22240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:47 dist-test-slave-pqr8 krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:47.928579 22240 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:37:47.929961 22240 webserver.cc:460] Webserver started at http://127.3.186.194:36573/ using document root <none> and password file <none>
I20260927 21:37:47.930522 22240 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:47.930616 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:47.930848 22240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:47.933409 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "6321412def6d432aada2870e1843fd3c"
format_stamp: "Formatted at 2026-09-27 21:37:47 on dist-test-slave-pqr8"
server_key: "892bd54c1a90804927e4452931742968"
server_key_iv: "29638c121e3cbfe70583f35f6790c07e"
server_key_version: "encryptionkey@0"
I20260927 21:37:47.934165 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "6321412def6d432aada2870e1843fd3c"
format_stamp: "Formatted at 2026-09-27 21:37:47 on dist-test-slave-pqr8"
server_key: "892bd54c1a90804927e4452931742968"
server_key_iv: "29638c121e3cbfe70583f35f6790c07e"
server_key_version: "encryptionkey@0"
I20260927 21:37:47.938735 22240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:37:47.941780 22255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:47.942924 22240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:47.943048 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "6321412def6d432aada2870e1843fd3c"
format_stamp: "Formatted at 2026-09-27 21:37:47 on dist-test-slave-pqr8"
server_key: "892bd54c1a90804927e4452931742968"
server_key_iv: "29638c121e3cbfe70583f35f6790c07e"
server_key_version: "encryptionkey@0"
I20260927 21:37:47.943192 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:47.961102 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:47.965648 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:47.965888 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:47.966852 22240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:47.968289 22240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:47.968374 22240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:47.968502 22240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:47.968560 22240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:48.000526 22240 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:46291
I20260927 21:37:48.000607 22367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:46291 every 8 connection(s)
I20260927 21:37:48.001658 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:48.008256 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22240
I20260927 21:37:48.008441 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:37:48.009020 3819 external_mini_cluster.cc:1428] Setting key a301ff6630baaa630dce6f031b5e0342
Sep 27 21:37:48 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545067, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:48.022148 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.004278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:53943 (local address 127.3.186.254:44287)
0927 21:37:48.004460 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.004465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.005392 (+ 927us) server_negotiation.cc:407] Connection header received
0927 21:37:48.007315 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.007320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.007408 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.007519 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:48.009382 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.010292 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.012369 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.012760 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.012850 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.016562 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:48.016567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:48.016570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:48.016598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:48.019388 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.020035 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.020042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.020045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.020180 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.020595 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.020601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.020605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.020867 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:48.021026 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.021628 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.021833 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:37:48.022934 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.004611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44287 (local address 127.3.186.194:53943)
0927 21:37:48.005249 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.005282 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.007047 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.007741 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.007749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.008435 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:48.009188 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.009213 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.010455 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.010459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.012230 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.012247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.012970 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.012979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.013289 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.013940 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:48.013973 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:48.016350 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:48.019518 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:48.019526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:48.019569 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:48.019916 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:48.020296 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:48.020301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:48.020305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:48.020486 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:48.020998 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:48.021005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:48.021287 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.021544 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.021975 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":264,"threads_started":1}
I20260927 21:37:48.024317 22368 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44287
I20260927 21:37:48.024585 22368 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:48.025303 22368 heartbeater.cc:511] Master 127.3.186.254:44287 requested a full tablet report, sending...
I20260927 21:37:48.026696 22045 ts_manager.cc:194] Registered new tserver with Master: 6321412def6d432aada2870e1843fd3c (127.3.186.194:46291)
I20260927 21:37:48.027467 22045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:53943
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:37:48.060652 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44287
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:48.372109 22373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:48.372340 22373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:48.372423 22373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:48.379216 22373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:48.379307 22373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:48.379408 22373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:48.388224 22373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:48.389206 22373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:48.390353 22373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:48.402215 22378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:48.402165 22381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:48.402251 22379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:48.402982 22373 server_base.cc:1034] running on GCE node
I20260927 21:37:48.403455 22373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:48.404202 22373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:48.405390 22373 hybrid_clock.cc:648] HybridClock initialized: now 1790545068405358 us; error 23 us; skew 500 ppm
I20260927 21:37:48.405725 22373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:48 dist-test-slave-pqr8 krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545068, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:48.408699 22373 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:37:48.409972 22373 webserver.cc:460] Webserver started at http://127.3.186.195:38751/ using document root <none> and password file <none>
I20260927 21:37:48.410526 22373 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:48.410629 22373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:48.410864 22373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:48.413343 22373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "3af65ea8ce4740698fcfc903e60e0e10"
format_stamp: "Formatted at 2026-09-27 21:37:48 on dist-test-slave-pqr8"
server_key: "744a36d2d7068401a4b90430bc847abf"
server_key_iv: "add8c0172b0b0d67771f27ed05ad8388"
server_key_version: "encryptionkey@0"
I20260927 21:37:48.414216 22373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "3af65ea8ce4740698fcfc903e60e0e10"
format_stamp: "Formatted at 2026-09-27 21:37:48 on dist-test-slave-pqr8"
server_key: "744a36d2d7068401a4b90430bc847abf"
server_key_iv: "add8c0172b0b0d67771f27ed05ad8388"
server_key_version: "encryptionkey@0"
I20260927 21:37:48.419232 22373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:37:48.422314 22388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:48.423449 22373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:48.423612 22373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "3af65ea8ce4740698fcfc903e60e0e10"
format_stamp: "Formatted at 2026-09-27 21:37:48 on dist-test-slave-pqr8"
server_key: "744a36d2d7068401a4b90430bc847abf"
server_key_iv: "add8c0172b0b0d67771f27ed05ad8388"
server_key_version: "encryptionkey@0"
I20260927 21:37:48.423791 22373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:48.434249 22373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:48.437729 22373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:48.437986 22373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:48.438863 22373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:48.440016 22373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:48.440114 22373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:48.440251 22373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:48.440321 22373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:48.471627 22373 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:33965
I20260927 21:37:48.471694 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:33965 every 8 connection(s)
I20260927 21:37:48.472697 22373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:48.479240 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22373
I20260927 21:37:48.479420 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:37:48.479820 3819 external_mini_cluster.cc:1428] Setting key 5e601cf8fd2cae2b8e932e1a96ae5095
Sep 27 21:37:48 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545068, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:48.491003 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.474992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:42759 (local address 127.3.186.254:44287)
0927 21:37:48.475156 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.475160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.476090 (+ 930us) server_negotiation.cc:407] Connection header received
0927 21:37:48.477831 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.477836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.477902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.477996 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:48.479295 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.480172 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.481787 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.482406 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.482535 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.485410 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:48.485420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:48.485423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:48.485452 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:48.488293 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.488967 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.488972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.488975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.489098 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.489475 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.489482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.489484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.489781 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:48.489923 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.490459 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.490670 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:37:48.491714 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.475338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:44287 (local address 127.3.186.195:42759)
0927 21:37:48.475923 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.475989 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.477520 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.478175 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.478181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.478633 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:48.479118 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.479135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.480338 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.480342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.481654 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.481666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.482626 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.482631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.482815 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.483345 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:48.483364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:48.485226 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:48.488441 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:48.488449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:48.488461 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:48.488835 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:48.489200 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:48.489204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:48.489208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:48.489367 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:48.489899 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:48.489905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:48.490160 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.490387 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.490836 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":248,"threads_started":1}
I20260927 21:37:48.492807 22501 heartbeater.cc:347] Connected to a master server at 127.3.186.254:44287
I20260927 21:37:48.493072 22501 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:48.493784 22501 heartbeater.cc:511] Master 127.3.186.254:44287 requested a full tablet report, sending...
I20260927 21:37:48.495105 22045 ts_manager.cc:194] Registered new tserver with Master: 3af65ea8ce4740698fcfc903e60e0e10 (127.3.186.195:33965)
I20260927 21:37:48.495921 22045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:42759
I20260927 21:37:48.506327 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:48.521660 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.510252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.3.186.254:44287)
0927 21:37:48.510434 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.510442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.510799 (+ 357us) server_negotiation.cc:407] Connection header received
0927 21:37:48.510906 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.510909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.510957 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.511045 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:48.512326 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.512902 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.514044 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.514377 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.514462 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.516627 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:48.516631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:48.516634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:48.516660 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:48.519109 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.519858 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.519861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.519863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.519967 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.520408 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.520416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.520418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.520668 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:48.520914 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.521242 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.521396 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:37:48.525381 22045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:48.531836 22045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:48.550024 22234 heartbeater.cc:503] Master 127.3.186.254:44287 was elected leader, sending a full tablet report...
I20260927 21:37:48.563158 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.543273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.3.186.194:46291)
0927 21:37:48.555567 (+ 12294us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.555578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.555607 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:37:48.555708 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.555718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.555959 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.556427 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:48.557203 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.558749 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.560680 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.562075 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.562220 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.562390 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.562714 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.562885 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12160,"thread_start_us":12114,"threads_started":1}
I20260927 21:37:48.563210 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.548595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:46291 (local address 127.0.0.1:54466)
0927 21:37:48.554152 (+ 5557us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.554171 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.554284 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.556179 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.556185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.556213 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:48.556957 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.556974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.558876 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.558882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.560527 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.560539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.562281 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.562287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.562457 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.562517 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.562661 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.562831 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5385,"mutex_wait_us":33,"thread_start_us":4102,"threads_started":1}
I20260927 21:37:48.564188 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.542527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35464)
0927 21:37:48.553485 (+ 10958us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.553541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.553755 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.557147 (+ 3392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.557151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.557172 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:48.557632 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.557645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.559425 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.559431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.561600 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.561615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.563620 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.563629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.563744 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.563765 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.563817 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.563891 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10760,"mutex_wait_us":30,"spinlock_wait_cycles":9472,"thread_start_us":11836,"threads_started":1}
I20260927 21:37:48.565068 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.542317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:33965 (local address 127.0.0.1:43726)
0927 21:37:48.553480 (+ 11163us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.553540 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.553755 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.558316 (+ 4561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.558321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.558345 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:48.558683 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.558695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.560628 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.560635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.562652 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.562664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.564306 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.564312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.564478 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.564510 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.564614 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.564734 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10866,"mutex_wait_us":2,"spinlock_wait_cycles":4352,"thread_start_us":349,"threads_started":1}
I20260927 21:37:48.565153 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.543065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.3.186.193:33765)
0927 21:37:48.556034 (+ 12969us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.556042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.556069 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:37:48.556158 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.556164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.556375 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.556569 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:48.557831 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.559288 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.561789 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.563092 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.563227 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.564740 (+ 1513us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.564796 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.564896 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12819,"thread_start_us":12666,"threads_started":1}
I20260927 21:37:48.565857 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.542988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.3.186.195:33965)
0927 21:37:48.557214 (+ 14226us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.557221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.557249 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:37:48.557343 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.557353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.558112 (+ 759us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.558853 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:48.559063 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.560482 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.562810 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.564092 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.564222 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.564422 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.565397 (+ 975us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.565572 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14047,"thread_start_us":13903,"threads_started":1}
I20260927 21:37:48.567142 22303 tablet_service.cc:1467] Processing CreateTablet for tablet 62b4dd420b254de1bfafe78bf630dc33 (DEFAULT_TABLE table=test-table [id=8a008d46b85b48cb963ae98fd0842d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:48.568603 22303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b4dd420b254de1bfafe78bf630dc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:48.568580 22436 tablet_service.cc:1467] Processing CreateTablet for tablet 62b4dd420b254de1bfafe78bf630dc33 (DEFAULT_TABLE table=test-table [id=8a008d46b85b48cb963ae98fd0842d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:48.569216 22169 tablet_service.cc:1467] Processing CreateTablet for tablet 62b4dd420b254de1bfafe78bf630dc33 (DEFAULT_TABLE table=test-table [id=8a008d46b85b48cb963ae98fd0842d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:48.569713 22436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b4dd420b254de1bfafe78bf630dc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:48.570456 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b4dd420b254de1bfafe78bf630dc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:48.581027 22519 tablet_bootstrap.cc:492] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Bootstrap starting.
I20260927 21:37:48.584451 22519 tablet_bootstrap.cc:654] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:48.585399 22519 log.cc:824] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:48.588126 22519 tablet_bootstrap.cc:492] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: No bootstrap required, opened a new log
I20260927 21:37:48.588364 22519 ts_tablet_manager.cc:1397] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 21:37:48.589702 22520 tablet_bootstrap.cc:492] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Bootstrap starting.
I20260927 21:37:48.593426 22521 tablet_bootstrap.cc:492] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Bootstrap starting.
I20260927 21:37:48.593495 22520 tablet_bootstrap.cc:654] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:48.594537 22520 log.cc:824] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:48.596732 22521 tablet_bootstrap.cc:654] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:48.597252 22520 tablet_bootstrap.cc:492] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: No bootstrap required, opened a new log
I20260927 21:37:48.596936 22519 raft_consensus.cc:348] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.597363 22519 raft_consensus.cc:374] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:48.597482 22519 raft_consensus.cc:729] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af65ea8ce4740698fcfc903e60e0e10, State: Initialized, Role: FOLLOWER
I20260927 21:37:48.597564 22520 ts_tablet_manager.cc:1397] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260927 21:37:48.597925 22521 log.cc:824] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:48.598104 22519 consensus_queue.cc:260] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.599172 22501 heartbeater.cc:503] Master 127.3.186.254:44287 was elected leader, sending a full tablet report...
I20260927 21:37:48.600001 22519 ts_tablet_manager.cc:1428] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 21:37:48.602702 22521 tablet_bootstrap.cc:492] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: No bootstrap required, opened a new log
I20260927 21:37:48.602955 22521 ts_tablet_manager.cc:1397] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 21:37:48.606274 22520 raft_consensus.cc:348] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.606559 22520 raft_consensus.cc:374] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:48.606623 22520 raft_consensus.cc:729] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6321412def6d432aada2870e1843fd3c, State: Initialized, Role: FOLLOWER
I20260927 21:37:48.607096 22520 consensus_queue.cc:260] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.608063 22368 heartbeater.cc:503] Master 127.3.186.254:44287 was elected leader, sending a full tablet report...
I20260927 21:37:48.608686 22520 ts_tablet_manager.cc:1428] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260927 21:37:48.611589 22521 raft_consensus.cc:348] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.611979 22521 raft_consensus.cc:374] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:48.612099 22521 raft_consensus.cc:729] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e52b198de624442bf185a8d5764062a, State: Initialized, Role: FOLLOWER
I20260927 21:37:48.612571 22521 consensus_queue.cc:260] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.613819 22521 ts_tablet_manager.cc:1428] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
W20260927 21:37:48.724269 22502 tablet.cc:2369] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:48.736244 22527 raft_consensus.cc:482] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:48.736474 22527 raft_consensus.cc:504] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.737962 22527 leader_election.cc:288] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3af65ea8ce4740698fcfc903e60e0e10 (127.3.186.195:33965), 6321412def6d432aada2870e1843fd3c (127.3.186.194:46291)
I20260927 21:37:48.745461 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.738533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:58603 (local address 127.3.186.194:46291)
0927 21:37:48.738796 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.738805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.739511 (+ 706us) server_negotiation.cc:407] Connection header received
0927 21:37:48.739702 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.739707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.739758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.739850 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:48.740533 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.741437 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.743930 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.744674 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.744783 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.744831 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.745095 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.745219 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:37:48.745560 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.738364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:46291 (local address 127.3.186.193:58603)
0927 21:37:48.739415 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.739428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.739565 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.739867 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.739871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.739906 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:48.740367 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.740380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.741597 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.741603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.743775 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.743786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.744836 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.744840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.744936 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.744958 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.745062 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.745175 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":118,"thread_start_us":754,"threads_started":1}
I20260927 21:37:48.746588 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.738138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:33965 (local address 127.3.186.193:58343)
0927 21:37:48.739280 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.739298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:48.739460 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:48.739774 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:48.739778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:48.739800 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:48.740130 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.740156 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.741571 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.741580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.743777 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:48.743788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.745943 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.745951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:48.746112 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.746133 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:37:48.746203 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:37:48.746275 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":156,"thread_start_us":688,"threads_started":1}
I20260927 21:37:48.746486 22323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b4dd420b254de1bfafe78bf630dc33" candidate_uuid: "3e52b198de624442bf185a8d5764062a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6321412def6d432aada2870e1843fd3c" is_pre_election: true
I20260927 21:37:48.746819 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.738323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:58343 (local address 127.3.186.195:33965)
0927 21:37:48.738571 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.738579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.739392 (+ 813us) server_negotiation.cc:407] Connection header received
0927 21:37:48.739596 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.739602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.739666 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.739756 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:48.740289 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.741417 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.744893 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.745698 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.745831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.745894 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.746257 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.746380 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:37:48.747015 22323 raft_consensus.cc:2454] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b198de624442bf185a8d5764062a in term 0.
I20260927 21:37:48.747673 22124 leader_election.cc:302] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b198de624442bf185a8d5764062a, 6321412def6d432aada2870e1843fd3c; no voters:
I20260927 21:37:48.748046 22527 raft_consensus.cc:2781] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:48.748189 22527 raft_consensus.cc:482] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:48.748100 22456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b4dd420b254de1bfafe78bf630dc33" candidate_uuid: "3e52b198de624442bf185a8d5764062a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af65ea8ce4740698fcfc903e60e0e10" is_pre_election: true
I20260927 21:37:48.748318 22527 raft_consensus.cc:3037] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:48.748456 22456 raft_consensus.cc:2454] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b198de624442bf185a8d5764062a in term 0.
I20260927 21:37:48.750639 22527 raft_consensus.cc:504] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.751238 22527 leader_election.cc:288] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [CANDIDATE]: Term 1 election: Requested vote from peers 3af65ea8ce4740698fcfc903e60e0e10 (127.3.186.195:33965), 6321412def6d432aada2870e1843fd3c (127.3.186.194:46291)
I20260927 21:37:48.751605 22456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b4dd420b254de1bfafe78bf630dc33" candidate_uuid: "3e52b198de624442bf185a8d5764062a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af65ea8ce4740698fcfc903e60e0e10"
I20260927 21:37:48.751688 22323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b4dd420b254de1bfafe78bf630dc33" candidate_uuid: "3e52b198de624442bf185a8d5764062a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6321412def6d432aada2870e1843fd3c"
I20260927 21:37:48.751812 22456 raft_consensus.cc:3037] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:48.751843 22323 raft_consensus.cc:3037] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:37:48.753736 22369 tablet.cc:2369] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:48.754904 22456 raft_consensus.cc:2454] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b198de624442bf185a8d5764062a in term 1.
I20260927 21:37:48.754949 22323 raft_consensus.cc:2454] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b198de624442bf185a8d5764062a in term 1.
I20260927 21:37:48.755316 22122 leader_election.cc:302] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af65ea8ce4740698fcfc903e60e0e10, 3e52b198de624442bf185a8d5764062a; no voters:
I20260927 21:37:48.755657 22527 raft_consensus.cc:2781] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:48.756047 22527 raft_consensus.cc:686] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 1 LEADER]: Becoming Leader. State: Replica: 3e52b198de624442bf185a8d5764062a, State: Running, Role: LEADER
I20260927 21:37:48.756481 22527 consensus_queue.cc:237] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } }
I20260927 21:37:48.760298 22045 catalog_manager.cc:5495] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e52b198de624442bf185a8d5764062a (127.3.186.193). New cstate: current_term: 1 leader_uuid: "3e52b198de624442bf185a8d5764062a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e52b198de624442bf185a8d5764062a" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 } health_report { overall_health: UNKNOWN } } }
W20260927 21:37:48.773753 22235 tablet.cc:2369] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:48.798019 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.792508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.3.186.193:33765)
0927 21:37:48.792678 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.792682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.792853 (+ 171us) server_negotiation.cc:407] Connection header received
0927 21:37:48.793012 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.793015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.793096 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.793207 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:48.793579 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.794267 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.795593 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.795960 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.796170 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.796835 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:48.797425 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:48.797624 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.797772 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.797815 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:37:48.805241 22456 raft_consensus.cc:1261] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 1 FOLLOWER]: Refusing update from remote peer 3e52b198de624442bf185a8d5764062a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:48.805241 22323 raft_consensus.cc:1261] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 1 FOLLOWER]: Refusing update from remote peer 3e52b198de624442bf185a8d5764062a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:48.806288 22527 consensus_queue.cc:1035] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6321412def6d432aada2870e1843fd3c" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:48.806569 22530 consensus_queue.cc:1035] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af65ea8ce4740698fcfc903e60e0e10" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:48.817045 22533 mvcc.cc:205] Tried to move back new op lower bound from 7334072601817014272 to 7334072601629790208. Current Snapshot: MvccSnapshot[applied={T|T < 7334072601817014272}]
I20260927 21:37:48.831146 22045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:48.831458 22045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:48.833623 22535 mvcc.cc:205] Tried to move back new op lower bound from 7334072601817014272 to 7334072601629790208. Current Snapshot: MvccSnapshot[applied={T|T < 7334072601817014272 or (T in {7334072601817014272})}]
I20260927 21:37:48.836189 22045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99d038e5c69b458b9eb19926a848d7ac: Sending DeleteTablet for 3 replicas of tablet 62b4dd420b254de1bfafe78bf630dc33
I20260927 21:37:48.837369 22436 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b4dd420b254de1bfafe78bf630dc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:48 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:43726
I20260927 21:37:48.837819 22169 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b4dd420b254de1bfafe78bf630dc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:48 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:35464
I20260927 21:37:48.839938 22303 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b4dd420b254de1bfafe78bf630dc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:48 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:54466
I20260927 21:37:48.841975 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22106
I20260927 21:37:48.846421 22543 tablet_replica.cc:330] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: stopping tablet replica
I20260927 21:37:48.846792 22543 raft_consensus.cc:2229] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:48.847339 22543 raft_consensus.cc:2258] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:48.849689 22543 ts_tablet_manager.cc:1905] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:48.850620 22545 tablet_replica.cc:330] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: stopping tablet replica
I20260927 21:37:48.850961 22545 raft_consensus.cc:2229] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:48.851357 22545 raft_consensus.cc:2258] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:48.853749 22545 ts_tablet_manager.cc:1905] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:48.856735 22544 tablet_replica.cc:330] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: stopping tablet replica
I20260927 21:37:48.857079 22544 raft_consensus.cc:2229] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:48.857673 22544 raft_consensus.cc:2258] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 21:37:48 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:37:48.861224 22543 ts_tablet_manager.cc:1918] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:48.861366 22543 log.cc:1196] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/62b4dd420b254de1bfafe78bf630dc33
I20260927 21:37:48.861790 22545 ts_tablet_manager.cc:1918] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:48.861869 22543 ts_tablet_manager.cc:1939] T 62b4dd420b254de1bfafe78bf630dc33 P 3af65ea8ce4740698fcfc903e60e0e10: Deleting consensus metadata
I20260927 21:37:48.861913 22545 log.cc:1196] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/62b4dd420b254de1bfafe78bf630dc33
I20260927 21:37:48.862411 22545 ts_tablet_manager.cc:1939] T 62b4dd420b254de1bfafe78bf630dc33 P 6321412def6d432aada2870e1843fd3c: Deleting consensus metadata
I20260927 21:37:48.863483 22029 catalog_manager.cc:4841] TS 3af65ea8ce4740698fcfc903e60e0e10 (127.3.186.195:33965): tablet 62b4dd420b254de1bfafe78bf630dc33 (table test-table [id=8a008d46b85b48cb963ae98fd0842d06]) successfully deleted
I20260927 21:37:48.863947 22031 catalog_manager.cc:4841] TS 6321412def6d432aada2870e1843fd3c (127.3.186.194:46291): tablet 62b4dd420b254de1bfafe78bf630dc33 (table test-table [id=8a008d46b85b48cb963ae98fd0842d06]) successfully deleted
I20260927 21:37:48.865096 22544 ts_tablet_manager.cc:1905] T 62b4dd420b254de1bfafe78bf630dc33 P 3e52b198de624442bf185a8d5764062a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:48.868096 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.843593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.3.186.193:33765)
0927 21:37:48.843752 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.843757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.848061 (+ 4304us) server_negotiation.cc:407] Connection header received
0927 21:37:48.848287 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.848292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.848385 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.848475 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:48.851719 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.852933 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.854864 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.855547 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.855641 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.861391 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:48.861405 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:48.861419 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:48.861464 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:48.865276 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.866012 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.866020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.866029 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.866178 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.866561 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.866565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.866567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.866877 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:48.867223 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.867486 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.867739 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:37:48.868745 22229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:48.962229 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22106
I20260927 21:37:48.979525 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22240
W20260927 21:37:48.979473 22032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:33765 (error 108)
I20260927 21:37:48.981202 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.980823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35482)
0927 21:37:48.980984 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:48.981086 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:37:48.981405 22032 catalog_manager.cc:4584] TS 3e52b198de624442bf185a8d5764062a (127.3.186.193:33765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62b4dd420b254de1bfafe78bf630dc33: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Sep 27 21:37:48 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:37:48.994076 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:48.980338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.3.186.194:46291)
0927 21:37:48.980521 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:48.980527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:48.980638 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:37:48.980842 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:48.980852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:48.980917 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:48.981028 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:48.982372 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.983089 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.984355 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:48.984722 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:48.984847 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:48.989000 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:48.989011 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:48.989024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:48.989058 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:48.991492 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.992098 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.992105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.992107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.992223 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:48.992597 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:48.992602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:48.992605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:48.992943 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:48.993241 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:48.993497 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:37:48.993705 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:37:48.994752 22363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:49.034929 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.034482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35490)
0927 21:37:49.034669 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:49.034761 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:37:49.035923 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.035558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35502)
0927 21:37:49.035716 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:49.035808 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 21:37:49.080839 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22240
I20260927 21:37:49.096347 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.095902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35516)
0927 21:37:49.096118 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:49.096202 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:37:49.097324 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.096985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35522)
0927 21:37:49.097125 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:49.097219 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:37:49.098104 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22373
Sep 27 21:37:49 dist-test-slave-pqr8 krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:37:49.112462 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.098838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.3.186.195:33965)
0927 21:37:49.099032 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:49.099036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:49.099052 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:37:49.099109 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:49.099113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:49.099181 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:49.099257 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:49.100656 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:49.101349 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:49.102579 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:49.102945 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:49.103083 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:49.107140 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:49.107166 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:49.107191 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:49.107237 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:49.110266 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:49.110822 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:49.110828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:49.110831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:49.110957 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:49.111271 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:49.111274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:49.111276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:49.111584 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:49.111781 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:49.112005 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 21:37:49.112149 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:37:49.113061 22496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:49.178028 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.177475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35528)
0927 21:37:49.177733 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:49.177837 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 21:37:49.179090 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.178677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33765 (local address 127.0.0.1:35542)
0927 21:37:49.178884 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:49.178968 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 21:37:49.204581 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22373
I20260927 21:37:49.220715 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22013
I20260927 21:37:49.221683 22075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:49.326527 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22013
2026-09-27T21:37:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4041 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T21:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:37:49Z Disabled control of system clock
I20260927 21:37:49.396287 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:32775
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:32775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:37:49.700644 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:49.700898 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:49.700977 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:49.707993 22555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:49.708091 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:49.708128 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:49.708184 22555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:49.708227 22555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:49.716884 22555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:32775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:32775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:49.718068 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:49.719655 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:49.730720 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:49.730844 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:49.730779 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:49.731846 22555 server_base.cc:1034] running on GCE node
I20260927 21:37:49.732465 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:49.733711 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:49.734901 22555 hybrid_clock.cc:648] HybridClock initialized: now 1790545069734869 us; error 29 us; skew 500 ppm
I20260927 21:37:49.735181 22555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:49.737255 22555 webserver.cc:460] Webserver started at http://127.3.186.254:42521/ using document root <none> and password file <none>
I20260927 21:37:49.737900 22555 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:49.737984 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:49.738237 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:49.740768 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "e7ad18e9115143249d27d34a0688364c"
format_stamp: "Formatted at 2026-09-27 21:37:49 on dist-test-slave-pqr8"
server_key: "9be97fbfd0624a126387fdbed58f3894"
server_key_iv: "c47d66e3ab9d07cc13ab792f4e91005d"
server_key_version: "encryptionkey@0"
I20260927 21:37:49.741439 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "e7ad18e9115143249d27d34a0688364c"
format_stamp: "Formatted at 2026-09-27 21:37:49 on dist-test-slave-pqr8"
server_key: "9be97fbfd0624a126387fdbed58f3894"
server_key_iv: "c47d66e3ab9d07cc13ab792f4e91005d"
server_key_version: "encryptionkey@0"
I20260927 21:37:49.745977 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 21:37:49.749038 22569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:49.750399 22555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:37:49.750567 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "e7ad18e9115143249d27d34a0688364c"
format_stamp: "Formatted at 2026-09-27 21:37:49 on dist-test-slave-pqr8"
server_key: "9be97fbfd0624a126387fdbed58f3894"
server_key_iv: "c47d66e3ab9d07cc13ab792f4e91005d"
server_key_version: "encryptionkey@0"
I20260927 21:37:49.750742 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:49.773973 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:49.774690 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:49.774966 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:49.795220 22555 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:32775
I20260927 21:37:49.795239 22620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:32775 every 8 connection(s)
I20260927 21:37:49.796396 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:49.799844 22621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:49.803880 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22555
I20260927 21:37:49.804066 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:37:49.804606 3819 external_mini_cluster.cc:1428] Setting key b1c35595fa48603849add794ffa512be
I20260927 21:37:49.810708 22621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c: Bootstrap starting.
I20260927 21:37:49.814000 22621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:49.815196 22621 log.cc:824] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:49.817970 22621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c: No bootstrap required, opened a new log
I20260927 21:37:49.818238 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:49.806419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.3.186.254:32775)
0927 21:37:49.807304 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:49.807314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:49.807354 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:37:49.809298 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:49.809319 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:49.809794 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:49.810086 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:49.810663 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:49.811934 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:49.813739 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:49.814447 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:49.814586 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:49.815863 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:49.815876 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:49.815898 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:49.816572 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:49.816640 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:49.816660 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:49.816788 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:49.816993 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 21:37:49.817286 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":333,"threads_started":1}
I20260927 21:37:49.825210 22621 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ad18e9115143249d27d34a0688364c" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 32775 } }
I20260927 21:37:49.825598 22621 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:49.825716 22621 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ad18e9115143249d27d34a0688364c, State: Initialized, Role: FOLLOWER
I20260927 21:37:49.826292 22621 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ad18e9115143249d27d34a0688364c" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 32775 } }
I20260927 21:37:49.826534 22621 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:49.826675 22621 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:49.826846 22621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:49.828898 22621 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ad18e9115143249d27d34a0688364c" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 32775 } }
I20260927 21:37:49.829493 22621 leader_election.cc:302] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ad18e9115143249d27d34a0688364c; no voters:
I20260927 21:37:49.830123 22621 leader_election.cc:288] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:37:49.830263 22626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:49.830946 22626 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [term 1 LEADER]: Becoming Leader. State: Replica: e7ad18e9115143249d27d34a0688364c, State: Running, Role: LEADER
I20260927 21:37:49.831523 22626 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ad18e9115143249d27d34a0688364c" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 32775 } }
I20260927 21:37:49.832031 22621 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:49.834074 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ad18e9115143249d27d34a0688364c. Latest consensus state: current_term: 1 leader_uuid: "e7ad18e9115143249d27d34a0688364c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ad18e9115143249d27d34a0688364c" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 32775 } } }
I20260927 21:37:49.834160 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ad18e9115143249d27d34a0688364c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ad18e9115143249d27d34a0688364c" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 32775 } } }
I20260927 21:37:49.834439 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:49.834574 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:49.836431 22632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:49.841725 22632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:49.849108 22632 catalog_manager.cc:1347] Generated new cluster ID: b0397bcf23914068832036fd10b9e0ae
I20260927 21:37:49.849210 22632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:49.857887 22632 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:37:49.859045 22632 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:49.866622 22632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c: Generated new TSK 0
I20260927 21:37:49.867274 22632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:37:49.876392 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:32775
--builtin_ntp_servers=127.3.186.212:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:37:50.190392 22645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:50.190639 22645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:50.190786 22645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:50.197651 22645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:50.197753 22645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:50.197930 22645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:50.206537 22645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:32775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:50.207648 22645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:50.209270 22645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:50.221382 22653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:50.221439 22651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:50.221406 22650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:50.222157 22645 server_base.cc:1034] running on GCE node
I20260927 21:37:50.222682 22645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:50.223506 22645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:50.224723 22645 hybrid_clock.cc:648] HybridClock initialized: now 1790545070224690 us; error 57 us; skew 500 ppm
I20260927 21:37:50.225010 22645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:50.227234 22645 webserver.cc:460] Webserver started at http://127.3.186.193:34313/ using document root <none> and password file <none>
I20260927 21:37:50.227860 22645 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:50.227986 22645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:50.228294 22645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:50.230911 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "bd9de6878b674e968b79fc7d722d8dbc"
format_stamp: "Formatted at 2026-09-27 21:37:50 on dist-test-slave-pqr8"
server_key: "bb639f9ea2d55973d9fd71ad3760e67a"
server_key_iv: "136dfb7213947d9af72f05355fe40947"
server_key_version: "encryptionkey@0"
I20260927 21:37:50.231704 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "bd9de6878b674e968b79fc7d722d8dbc"
format_stamp: "Formatted at 2026-09-27 21:37:50 on dist-test-slave-pqr8"
server_key: "bb639f9ea2d55973d9fd71ad3760e67a"
server_key_iv: "136dfb7213947d9af72f05355fe40947"
server_key_version: "encryptionkey@0"
I20260927 21:37:50.236351 22645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 21:37:50.239758 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:50.240934 22645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:37:50.241106 22645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "bd9de6878b674e968b79fc7d722d8dbc"
format_stamp: "Formatted at 2026-09-27 21:37:50 on dist-test-slave-pqr8"
server_key: "bb639f9ea2d55973d9fd71ad3760e67a"
server_key_iv: "136dfb7213947d9af72f05355fe40947"
server_key_version: "encryptionkey@0"
I20260927 21:37:50.241307 22645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:50.255725 22645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:50.256407 22645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:50.256681 22645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:50.257634 22645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:50.258910 22645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:50.259025 22645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:50.259156 22645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:50.259243 22645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:50.290525 22645 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:37279
I20260927 21:37:50.290601 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:37279 every 8 connection(s)
I20260927 21:37:50.291702 22645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:50.297670 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22645
I20260927 21:37:50.297833 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:37:50.298254 3819 external_mini_cluster.cc:1428] Setting key 9149b5b488ff7359f3d75b871d4acc50
I20260927 21:37:50.300668 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:32775
--builtin_ntp_servers=127.3.186.212:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:37:50.303494 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:50.293976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:60335 (local address 127.3.186.254:32775)
0927 21:37:50.294188 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:50.294194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:50.295403 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 21:37:50.297004 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:50.297007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:50.297072 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:50.297153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:50.298306 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.299134 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.300702 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.301262 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.301396 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:50.302737 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:50.302743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:50.302748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:50.302774 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:50.302803 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:50.302815 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:50.302906 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:50.303118 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 21:37:50.303208 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:37:50.304440 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:50.294461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:32775 (local address 127.3.186.193:60335)
0927 21:37:50.295256 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:50.295290 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:50.296772 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:50.297354 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:50.297361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:50.297439 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:50.298143 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:50.298160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.299296 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.299301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:50.300565 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:50.300577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.301497 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.301534 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:50.301749 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:50.302427 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:50.302429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:50.302464 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:50.302536 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:50.302906 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:50.302919 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:50.302929 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:37:50.303098 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 21:37:50.303435 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":248,"threads_started":1}
I20260927 21:37:50.305948 22772 heartbeater.cc:347] Connected to a master server at 127.3.186.254:32775
I20260927 21:37:50.306252 22772 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:50.306926 22772 heartbeater.cc:511] Master 127.3.186.254:32775 requested a full tablet report, sending...
I20260927 21:37:50.308914 22586 ts_manager.cc:194] Registered new tserver with Master: bd9de6878b674e968b79fc7d722d8dbc (127.3.186.193:37279)
I20260927 21:37:50.310555 22586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.193:60335
W20260927 21:37:50.609309 22775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:50.609602 22775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:50.609738 22775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:50.616658 22775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:50.616752 22775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:50.616868 22775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:50.625411 22775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:32775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:50.626535 22775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:50.628132 22775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:50.640410 22780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:50.640492 22781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:50.640545 22783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:50.641762 22775 server_base.cc:1034] running on GCE node
I20260927 21:37:50.642369 22775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:50.643209 22775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:50.644418 22775 hybrid_clock.cc:648] HybridClock initialized: now 1790545070644368 us; error 42 us; skew 500 ppm
I20260927 21:37:50.644727 22775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:50.652966 22775 webserver.cc:460] Webserver started at http://127.3.186.194:34629/ using document root <none> and password file <none>
I20260927 21:37:50.653656 22775 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:50.653775 22775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:50.654137 22775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:50.656800 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "c1e83158a295453abb27e44d9ceee812"
format_stamp: "Formatted at 2026-09-27 21:37:50 on dist-test-slave-pqr8"
server_key: "dc35e8abc115b47d93bcca25262dc3ac"
server_key_iv: "c9f82356da8fdf510cf9cb0b89905161"
server_key_version: "encryptionkey@0"
I20260927 21:37:50.657634 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "c1e83158a295453abb27e44d9ceee812"
format_stamp: "Formatted at 2026-09-27 21:37:50 on dist-test-slave-pqr8"
server_key: "dc35e8abc115b47d93bcca25262dc3ac"
server_key_iv: "c9f82356da8fdf510cf9cb0b89905161"
server_key_version: "encryptionkey@0"
I20260927 21:37:50.662233 22775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:37:50.665236 22789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:50.666509 22775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:37:50.666684 22775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "c1e83158a295453abb27e44d9ceee812"
format_stamp: "Formatted at 2026-09-27 21:37:50 on dist-test-slave-pqr8"
server_key: "dc35e8abc115b47d93bcca25262dc3ac"
server_key_iv: "c9f82356da8fdf510cf9cb0b89905161"
server_key_version: "encryptionkey@0"
I20260927 21:37:50.666884 22775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:50.701371 22775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:50.702100 22775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:50.702385 22775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:50.703305 22775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:50.704586 22775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:50.704698 22775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:50.704839 22775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:50.704958 22775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:50.736951 22775 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:44417
I20260927 21:37:50.737021 22901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:44417 every 8 connection(s)
I20260927 21:37:50.738236 22775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:50.744015 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22775
I20260927 21:37:50.744192 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:37:50.744606 3819 external_mini_cluster.cc:1428] Setting key f61fc281eb3f9e57b996e00f0c07e986
I20260927 21:37:50.747151 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:32775
--builtin_ntp_servers=127.3.186.212:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:37:50.750984 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:50.740553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:50949 (local address 127.3.186.254:32775)
0927 21:37:50.740732 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:50.740736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:50.742051 (+ 1315us) server_negotiation.cc:407] Connection header received
0927 21:37:50.743767 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:50.743777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:50.743869 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:50.743988 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:50.745235 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.746086 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.747691 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.748467 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.748622 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:50.750077 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:50.750084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:50.750092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:50.750116 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:50.750169 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:50.750186 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:50.750276 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:50.750484 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 21:37:50.750601 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:37:50.751698 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:50.741021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:32775 (local address 127.3.186.194:50949)
0927 21:37:50.741894 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:50.741942 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:50.743481 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:50.744311 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:50.744319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:50.744417 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:50.745027 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:50.745046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.746284 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.746288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:50.747550 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:50.747561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:50.748671 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:50.748676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:50.748856 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:50.749688 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:50.749695 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:50.749755 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:50.749868 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:50.750286 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:50.750294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:50.750299 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:37:50.750432 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 21:37:50.750702 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":261,"threads_started":1}
I20260927 21:37:50.753094 22902 heartbeater.cc:347] Connected to a master server at 127.3.186.254:32775
I20260927 21:37:50.753425 22902 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:50.754125 22902 heartbeater.cc:511] Master 127.3.186.254:32775 requested a full tablet report, sending...
I20260927 21:37:50.755584 22586 ts_manager.cc:194] Registered new tserver with Master: c1e83158a295453abb27e44d9ceee812 (127.3.186.194:44417)
I20260927 21:37:50.756636 22586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.194:50949
W20260927 21:37:51.052753 22905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:51.052990 22905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:51.053135 22905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:51.060380 22905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:51.060462 22905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:51.060626 22905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:51.069329 22905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:32775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:51.070416 22905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:51.072050 22905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:51.084074 22910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:51.084039 22911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:51.084009 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:51.084757 22905 server_base.cc:1034] running on GCE node
I20260927 21:37:51.085309 22905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:51.086225 22905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:51.087453 22905 hybrid_clock.cc:648] HybridClock initialized: now 1790545071087421 us; error 50 us; skew 500 ppm
I20260927 21:37:51.087749 22905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:51.089928 22905 webserver.cc:460] Webserver started at http://127.3.186.195:43509/ using document root <none> and password file <none>
I20260927 21:37:51.090543 22905 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:51.090667 22905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:51.090966 22905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:51.093500 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "97e5115f8d21470db81605bb7e6c0598"
format_stamp: "Formatted at 2026-09-27 21:37:51 on dist-test-slave-pqr8"
server_key: "1fb43507af8bfc5ad3c1cfeeef0e4fa2"
server_key_iv: "c9eb981dff1abdaf9b73e84667ee8a79"
server_key_version: "encryptionkey@0"
I20260927 21:37:51.094312 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "97e5115f8d21470db81605bb7e6c0598"
format_stamp: "Formatted at 2026-09-27 21:37:51 on dist-test-slave-pqr8"
server_key: "1fb43507af8bfc5ad3c1cfeeef0e4fa2"
server_key_iv: "c9eb981dff1abdaf9b73e84667ee8a79"
server_key_version: "encryptionkey@0"
I20260927 21:37:51.098877 22905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:37:51.101922 22919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:51.103081 22905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:37:51.103257 22905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "97e5115f8d21470db81605bb7e6c0598"
format_stamp: "Formatted at 2026-09-27 21:37:51 on dist-test-slave-pqr8"
server_key: "1fb43507af8bfc5ad3c1cfeeef0e4fa2"
server_key_iv: "c9eb981dff1abdaf9b73e84667ee8a79"
server_key_version: "encryptionkey@0"
I20260927 21:37:51.103467 22905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:51.133255 22905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:51.134011 22905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:51.134280 22905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:51.135133 22905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:51.136451 22905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:51.136579 22905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:51.136693 22905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:51.136751 22905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:51.169852 22905 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:38485
I20260927 21:37:51.169929 23031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:38485 every 8 connection(s)
I20260927 21:37:51.171043 22905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:51.180089 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 22905
I20260927 21:37:51.180261 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:37:51.180735 3819 external_mini_cluster.cc:1428] Setting key 359e1f2d85a1d670f9ebe5c4c5246588
I20260927 21:37:51.183164 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.173194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:58495 (local address 127.3.186.254:32775)
0927 21:37:51.173370 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.173374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.174707 (+ 1333us) server_negotiation.cc:407] Connection header received
0927 21:37:51.176477 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.176482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.176573 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.176685 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:51.177783 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.178510 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.180336 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.180745 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.180891 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.182373 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:51.182379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:51.182394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:51.182410 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:51.182439 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:51.182452 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:51.182547 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.182763 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.182855 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:37:51.184032 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.173766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:32775 (local address 127.3.186.195:58495)
0927 21:37:51.174566 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.174604 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:51.176194 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:51.176911 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:51.176919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:51.177016 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:51.177603 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.177622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.178714 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.178721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.180125 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.180147 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.180969 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.180976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.181287 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.182075 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:51.182078 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:51.182113 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:51.182178 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:51.182545 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:51.182557 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:51.182566 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:37:51.182734 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 21:37:51.183070 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":249,"threads_started":1}
I20260927 21:37:51.185214 23032 heartbeater.cc:347] Connected to a master server at 127.3.186.254:32775
I20260927 21:37:51.185482 23032 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:51.186204 23032 heartbeater.cc:511] Master 127.3.186.254:32775 requested a full tablet report, sending...
I20260927 21:37:51.187425 22586 ts_manager.cc:194] Registered new tserver with Master: 97e5115f8d21470db81605bb7e6c0598 (127.3.186.195:38485)
I20260927 21:37:51.188138 22586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.195:58495
I20260927 21:37:51.195143 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:51.204267 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.199072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.3.186.254:32775)
0927 21:37:51.199220 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.199224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.199641 (+ 417us) server_negotiation.cc:407] Connection header received
0927 21:37:51.199763 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.199766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.199818 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.199892 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:51.200294 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.200971 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.202292 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.202655 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.202744 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.203642 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:51.203646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:51.203649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:51.203664 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:51.203690 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:51.203700 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:51.203818 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.204031 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.204069 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:37:51.207662 22586 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:51.214290 22586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:51.244907 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.224764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.3.186.195:38485)
0927 21:37:51.236933 (+ 12169us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.236944 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.236985 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:37:51.237111 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.237123 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.237361 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.237619 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:51.238317 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.239984 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.242700 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.243706 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.243858 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.244158 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.244435 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.244619 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11924,"thread_start_us":11766,"threads_started":1}
I20260927 21:37:51.244952 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.224568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:38485 (local address 127.0.0.1:57950)
0927 21:37:51.234461 (+ 9893us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.234500 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:51.234674 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:51.237622 (+ 2948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:51.237630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:51.237678 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:51.238087 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.238100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.240174 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.240180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.242474 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.242489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.243890 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.243897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.244142 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.244198 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:37:51.244327 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:37:51.244584 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9581,"mutex_wait_us":60,"spinlock_wait_cycles":13184,"thread_start_us":12677,"threads_started":1}
I20260927 21:37:51.245834 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.225650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.3.186.194:44417)
0927 21:37:51.237114 (+ 11464us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.237120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.237573 (+ 453us) server_negotiation.cc:407] Connection header received
0927 21:37:51.237870 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.237878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.238113 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.238268 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:51.239229 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.240659 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.243089 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.244504 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.244621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.244855 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.245333 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.245491 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11308,"thread_start_us":11172,"threads_started":1}
I20260927 21:37:51.246591 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.224852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:44417 (local address 127.0.0.1:41718)
0927 21:37:51.237422 (+ 12570us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.237435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:51.237669 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:51.238591 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:51.238595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:51.238619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:51.239048 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.239061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.240795 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.240800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.242928 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.242944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.244990 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.244997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.245153 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.245182 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:37:51.246086 (+ 904us) client_negotiation.cc:240] Negotiation successful
0927 21:37:51.246234 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12482,"mutex_wait_us":1,"thread_start_us":10017,"threads_started":1}
I20260927 21:37:51.248796 22967 tablet_service.cc:1467] Processing CreateTablet for tablet 8aeac4999e5542d498db1f38eb9118e6 (DEFAULT_TABLE table=test-table [id=809064f3e6024ff19e1627d677c1445c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:51.249910 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.237833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.3.186.193:37279)
0927 21:37:51.239678 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.239688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.240205 (+ 517us) server_negotiation.cc:407] Connection header received
0927 21:37:51.240415 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.240422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.240613 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.240789 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:51.242246 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.243762 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.246196 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.247418 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.247499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.247638 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.249404 (+ 1766us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.249595 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":264,"threads_started":1}
I20260927 21:37:51.250124 22967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aeac4999e5542d498db1f38eb9118e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:51.249825 22837 tablet_service.cc:1467] Processing CreateTablet for tablet 8aeac4999e5542d498db1f38eb9118e6 (DEFAULT_TABLE table=test-table [id=809064f3e6024ff19e1627d677c1445c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:51.250540 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.238405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33440)
0927 21:37:51.240095 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.240107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:51.240242 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:51.241612 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:51.241616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:51.241634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:51.242039 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.242051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.243933 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.243938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.245993 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.246008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.247747 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.247751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.247863 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.247882 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 21:37:51.250158 (+ 2276us) client_negotiation.cc:240] Negotiation successful
0927 21:37:51.250276 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"thread_start_us":1459,"threads_started":1}
I20260927 21:37:51.251214 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aeac4999e5542d498db1f38eb9118e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:51.253728 22707 tablet_service.cc:1467] Processing CreateTablet for tablet 8aeac4999e5542d498db1f38eb9118e6 (DEFAULT_TABLE table=test-table [id=809064f3e6024ff19e1627d677c1445c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:51.255064 22707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aeac4999e5542d498db1f38eb9118e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:51.261682 23050 tablet_bootstrap.cc:492] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Bootstrap starting.
I20260927 21:37:51.264027 23051 tablet_bootstrap.cc:492] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Bootstrap starting.
I20260927 21:37:51.265174 23050 tablet_bootstrap.cc:654] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:51.266511 23050 log.cc:824] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:51.267865 23051 tablet_bootstrap.cc:654] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:51.269011 23051 log.cc:824] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:51.269501 23050 tablet_bootstrap.cc:492] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: No bootstrap required, opened a new log
I20260927 21:37:51.269822 23050 ts_tablet_manager.cc:1397] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260927 21:37:51.270164 23053 tablet_bootstrap.cc:492] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Bootstrap starting.
I20260927 21:37:51.273270 23053 tablet_bootstrap.cc:654] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:51.274341 23053 log.cc:824] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:51.274544 23051 tablet_bootstrap.cc:492] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: No bootstrap required, opened a new log
I20260927 21:37:51.274783 23051 ts_tablet_manager.cc:1397] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260927 21:37:51.277101 23053 tablet_bootstrap.cc:492] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: No bootstrap required, opened a new log
I20260927 21:37:51.277366 23053 ts_tablet_manager.cc:1397] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 21:37:51.279546 23050 raft_consensus.cc:348] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.279951 23050 raft_consensus.cc:374] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:51.280047 23050 raft_consensus.cc:729] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e83158a295453abb27e44d9ceee812, State: Initialized, Role: FOLLOWER
I20260927 21:37:51.280627 23050 consensus_queue.cc:260] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.281652 22902 heartbeater.cc:503] Master 127.3.186.254:32775 was elected leader, sending a full tablet report...
I20260927 21:37:51.282251 23050 ts_tablet_manager.cc:1428] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260927 21:37:51.284232 23051 raft_consensus.cc:348] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.284592 23051 raft_consensus.cc:374] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:51.284696 23051 raft_consensus.cc:729] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e5115f8d21470db81605bb7e6c0598, State: Initialized, Role: FOLLOWER
I20260927 21:37:51.285239 23051 consensus_queue.cc:260] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.285969 23053 raft_consensus.cc:348] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.286321 23032 heartbeater.cc:503] Master 127.3.186.254:32775 was elected leader, sending a full tablet report...
I20260927 21:37:51.286355 23053 raft_consensus.cc:374] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:51.286481 23053 raft_consensus.cc:729] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9de6878b674e968b79fc7d722d8dbc, State: Initialized, Role: FOLLOWER
I20260927 21:37:51.286983 23051 ts_tablet_manager.cc:1428] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260927 21:37:51.287144 23053 consensus_queue.cc:260] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.288218 22772 heartbeater.cc:503] Master 127.3.186.254:32775 was elected leader, sending a full tablet report...
I20260927 21:37:51.288735 23053 ts_tablet_manager.cc:1428] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260927 21:37:51.293792 22773 tablet.cc:2369] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:51.374717 23057 raft_consensus.cc:482] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:51.375023 23057 raft_consensus.cc:504] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.376457 23057 leader_election.cc:288] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e83158a295453abb27e44d9ceee812 (127.3.186.194:44417), bd9de6878b674e968b79fc7d722d8dbc (127.3.186.193:37279)
I20260927 21:37:51.382682 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.376853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:51703 (local address 127.3.186.194:44417)
0927 21:37:51.377062 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.377067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.377089 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:37:51.377379 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.377387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.377472 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.377603 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:51.378052 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.379023 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.380820 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.381832 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.382020 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.382091 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.382326 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.382439 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:37:51.382712 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.376709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:44417 (local address 127.3.186.195:51703)
0927 21:37:51.377010 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.377032 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:51.377164 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:51.377646 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:51.377650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:51.377672 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:51.377867 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.377898 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.379157 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.379163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.380660 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.380671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.381988 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.381994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.382178 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.382199 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:37:51.382301 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:37:51.382431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":21}
I20260927 21:37:51.383337 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.376967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:59397 (local address 127.3.186.193:37279)
0927 21:37:51.377133 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.377140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.377748 (+ 608us) server_negotiation.cc:407] Connection header received
0927 21:37:51.377965 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.377972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.378068 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.378135 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:51.378737 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.379560 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.381432 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.382335 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.382508 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.382569 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.383027 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.383166 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:37:51.383486 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.376807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.3.186.195:59397)
0927 21:37:51.377638 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.377656 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:51.377825 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:51.378180 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:51.378186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:51.378211 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:51.378595 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.378606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.379700 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.379706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.381241 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:51.381255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.382499 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.382505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:51.382673 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.382694 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:37:51.382974 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 21:37:51.383098 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":3,"spinlock_wait_cycles":37248,"thread_start_us":483,"threads_started":1}
I20260927 21:37:51.383634 22857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aeac4999e5542d498db1f38eb9118e6" candidate_uuid: "97e5115f8d21470db81605bb7e6c0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e83158a295453abb27e44d9ceee812" is_pre_election: true
I20260927 21:37:51.384218 22857 raft_consensus.cc:2454] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e5115f8d21470db81605bb7e6c0598 in term 0.
I20260927 21:37:51.384482 22727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aeac4999e5542d498db1f38eb9118e6" candidate_uuid: "97e5115f8d21470db81605bb7e6c0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd9de6878b674e968b79fc7d722d8dbc" is_pre_election: true
I20260927 21:37:51.384881 22921 leader_election.cc:302] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e5115f8d21470db81605bb7e6c0598, c1e83158a295453abb27e44d9ceee812; no voters:
I20260927 21:37:51.384960 22727 raft_consensus.cc:2454] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e5115f8d21470db81605bb7e6c0598 in term 0.
I20260927 21:37:51.385258 23057 raft_consensus.cc:2781] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:51.385455 23057 raft_consensus.cc:482] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:51.385641 23057 raft_consensus.cc:3037] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:51.387950 23057 raft_consensus.cc:504] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.388592 23057 leader_election.cc:288] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [CANDIDATE]: Term 1 election: Requested vote from peers c1e83158a295453abb27e44d9ceee812 (127.3.186.194:44417), bd9de6878b674e968b79fc7d722d8dbc (127.3.186.193:37279)
I20260927 21:37:51.388864 22857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aeac4999e5542d498db1f38eb9118e6" candidate_uuid: "97e5115f8d21470db81605bb7e6c0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e83158a295453abb27e44d9ceee812"
I20260927 21:37:51.389065 22857 raft_consensus.cc:3037] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:51.389073 22727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aeac4999e5542d498db1f38eb9118e6" candidate_uuid: "97e5115f8d21470db81605bb7e6c0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd9de6878b674e968b79fc7d722d8dbc"
I20260927 21:37:51.389251 22727 raft_consensus.cc:3037] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:51.391314 22857 raft_consensus.cc:2454] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e5115f8d21470db81605bb7e6c0598 in term 1.
I20260927 21:37:51.391418 22727 raft_consensus.cc:2454] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e5115f8d21470db81605bb7e6c0598 in term 1.
I20260927 21:37:51.391635 22921 leader_election.cc:302] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e5115f8d21470db81605bb7e6c0598, c1e83158a295453abb27e44d9ceee812; no voters:
I20260927 21:37:51.391866 23057 raft_consensus.cc:2781] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:51.392230 23057 raft_consensus.cc:686] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 1 LEADER]: Becoming Leader. State: Replica: 97e5115f8d21470db81605bb7e6c0598, State: Running, Role: LEADER
I20260927 21:37:51.392659 23057 consensus_queue.cc:237] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } }
I20260927 21:37:51.396531 22585 catalog_manager.cc:5495] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e5115f8d21470db81605bb7e6c0598 (127.3.186.195). New cstate: current_term: 1 leader_uuid: "97e5115f8d21470db81605bb7e6c0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e5115f8d21470db81605bb7e6c0598" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 } health_report { overall_health: UNKNOWN } } }
W20260927 21:37:51.422571 23033 tablet.cc:2369] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:51.435657 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.429717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.3.186.195:38485)
0927 21:37:51.429911 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.429918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.430042 (+ 124us) server_negotiation.cc:407] Connection header received
0927 21:37:51.430281 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.430286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.430367 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.430530 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:51.430913 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.431774 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.433361 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.433822 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.433953 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.434637 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:51.435087 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:51.435265 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.435423 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.435465 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:37:51.442998 22857 raft_consensus.cc:1261] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 1 FOLLOWER]: Refusing update from remote peer 97e5115f8d21470db81605bb7e6c0598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:51.443041 22727 raft_consensus.cc:1261] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 1 FOLLOWER]: Refusing update from remote peer 97e5115f8d21470db81605bb7e6c0598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:51.443861 23057 consensus_queue.cc:1035] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd9de6878b674e968b79fc7d722d8dbc" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:51.444096 23060 consensus_queue.cc:1035] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e83158a295453abb27e44d9ceee812" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:51.454691 23063 mvcc.cc:205] Tried to move back new op lower bound from 7334072612621864960 to 7334072612427972608. Current Snapshot: MvccSnapshot[applied={T|T < 7334072612621864960}]
I20260927 21:37:51.468488 22585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:51.468716 22585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:51.472085 22585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7ad18e9115143249d27d34a0688364c: Sending DeleteTablet for 3 replicas of tablet 8aeac4999e5542d498db1f38eb9118e6
I20260927 21:37:51.473042 22837 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aeac4999e5542d498db1f38eb9118e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:51 UTC) from {username='slave'} at 127.0.0.1:41718
I20260927 21:37:51.473126 22967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aeac4999e5542d498db1f38eb9118e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:51 UTC) from {username='slave'} at 127.0.0.1:57950
I20260927 21:37:51.473157 22707 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aeac4999e5542d498db1f38eb9118e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:51 UTC) from {username='slave'} at 127.0.0.1:33440
I20260927 21:37:51.473855 23073 tablet_replica.cc:330] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: stopping tablet replica
I20260927 21:37:51.474268 23073 raft_consensus.cc:2229] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:51.474766 23073 raft_consensus.cc:2258] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:51.474881 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22645
I20260927 21:37:51.476758 23073 ts_tablet_manager.cc:1905] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:51.485112 23073 ts_tablet_manager.cc:1918] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:51.485363 23073 log.cc:1196] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/8aeac4999e5542d498db1f38eb9118e6
I20260927 21:37:51.486107 23073 ts_tablet_manager.cc:1939] T 8aeac4999e5542d498db1f38eb9118e6 P c1e83158a295453abb27e44d9ceee812: Deleting consensus metadata
I20260927 21:37:51.487784 22571 catalog_manager.cc:4841] TS c1e83158a295453abb27e44d9ceee812 (127.3.186.194:44417): tablet 8aeac4999e5542d498db1f38eb9118e6 (table test-table [id=809064f3e6024ff19e1627d677c1445c]) successfully deleted
I20260927 21:37:51.488802 23074 tablet_replica.cc:330] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: stopping tablet replica
I20260927 21:37:51.489161 23074 raft_consensus.cc:2229] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:51.489584 23074 raft_consensus.cc:2258] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:51.493069 23074 ts_tablet_manager.cc:1905] T 8aeac4999e5542d498db1f38eb9118e6 P bd9de6878b674e968b79fc7d722d8dbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:51.494544 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.476178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.3.186.193:37279)
0927 21:37:51.476359 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.476365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.486810 (+ 10445us) server_negotiation.cc:407] Connection header received
0927 21:37:51.487181 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.487187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.487290 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.487362 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:51.488367 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.489606 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.490896 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.491535 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.491680 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.492991 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:51.493001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:51.493013 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:51.493638 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:51.493699 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:51.493721 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:51.493794 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.494086 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.494189 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:37:51.495764 22767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:51.499980 23075 tablet_replica.cc:330] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: stopping tablet replica
I20260927 21:37:51.500279 23075 raft_consensus.cc:2229] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:51.500722 23075 raft_consensus.cc:2258] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:51.502414 23075 ts_tablet_manager.cc:1905] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:51.510232 23075 ts_tablet_manager.cc:1918] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:51.510396 23075 log.cc:1196] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/8aeac4999e5542d498db1f38eb9118e6
I20260927 21:37:51.510967 23075 ts_tablet_manager.cc:1939] T 8aeac4999e5542d498db1f38eb9118e6 P 97e5115f8d21470db81605bb7e6c0598: Deleting consensus metadata
I20260927 21:37:51.512679 22573 catalog_manager.cc:4841] TS 97e5115f8d21470db81605bb7e6c0598 (127.3.186.195:38485): tablet 8aeac4999e5542d498db1f38eb9118e6 (table test-table [id=809064f3e6024ff19e1627d677c1445c]) successfully deleted
I20260927 21:37:51.584388 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22645
I20260927 21:37:51.607383 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22775
W20260927 21:37:51.607398 22573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:37279 (error 108)
I20260927 21:37:51.610111 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.609400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33462)
0927 21:37:51.609790 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.609946 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
W20260927 21:37:51.610455 22573 catalog_manager.cc:4584] TS bd9de6878b674e968b79fc7d722d8dbc (127.3.186.193:37279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8aeac4999e5542d498db1f38eb9118e6: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
I20260927 21:37:51.614889 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.608581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.3.186.194:44417)
0927 21:37:51.608757 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.608763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.608782 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:37:51.608845 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.608851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.608962 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.609072 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:51.609447 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.610514 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.611881 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.612293 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.612437 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.613479 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:51.613487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:51.613500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:51.614161 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:51.614225 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:51.614252 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:51.614366 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.614499 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.614598 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:37:51.615556 22897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:51.664398 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.663701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33464)
0927 21:37:51.664035 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.664169 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 21:37:51.665424 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.665056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33474)
0927 21:37:51.665213 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.665294 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:37:51.701303 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22775
I20260927 21:37:51.718262 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22905
I20260927 21:37:51.726025 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.719126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.3.186.195:38485)
0927 21:37:51.719315 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:51.719322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:51.719345 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:37:51.719408 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:51.719414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:51.719532 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:51.719647 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:51.719998 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.721193 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.722619 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:51.723284 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:51.723416 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:51.724416 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:51.724426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:51.724440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:51.725187 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:51.725247 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:51.725274 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:51.725375 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:51.725594 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:37:51.725677 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:37:51.726032 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.725353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33486)
0927 21:37:51.725710 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.725806 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 21:37:51.727145 23027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:51.727614 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.727042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33488)
0927 21:37:51.727255 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.727380 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:37:51.808137 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.807720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33498)
0927 21:37:51.807902 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.807999 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:37:51.809165 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:51.808808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:37279 (local address 127.0.0.1:33500)
0927 21:37:51.808999 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:51.809063 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:37279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 21:37:51.852885 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22905
I20260927 21:37:51.872257 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 22555
I20260927 21:37:51.873292 22616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:51.979131 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 22555
2026-09-27T21:37:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2651 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:37:52 dist-test-slave-pqr8 krb5kdc[23084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:37:52 dist-test-slave-pqr8 krb5kdc[23084](info): set up 2 sockets
Sep 27 21:37:52 dist-test-slave-pqr8 krb5kdc[23084](info): commencing operation
krb5kdc: starting...
W20260927 21:37:53.050019 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:37:53 dist-test-slave-pqr8 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:37:53.260032 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46063
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:46063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:53.562592 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:53.562896 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:53.563082 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:53.570154 23097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:53.570283 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:53.570363 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:53.570442 23097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:53.570520 23097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:53.579522 23097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:46063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:53.580651 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:53.581837 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:53.593225 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:53.593353 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:53.593333 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:53.594527 23097 server_base.cc:1034] running on GCE node
I20260927 21:37:53.595216 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:53.596478 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:53.597730 23097 hybrid_clock.cc:648] HybridClock initialized: now 1790545073597683 us; error 47 us; skew 500 ppm
I20260927 21:37:53.598074 23097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:53 dist-test-slave-pqr8 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545073, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:53.601032 23097 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:37:53.602428 23097 webserver.cc:460] Webserver started at http://127.3.186.254:36157/ using document root <none> and password file <none>
I20260927 21:37:53.603072 23097 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:53.603214 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:53.603518 23097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:53.606285 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "cd766495cb564398be0c38208ac7dbc6"
format_stamp: "Formatted at 2026-09-27 21:37:53 on dist-test-slave-pqr8"
server_key: "927d863a0165d312b10b8b6d4138d44a"
server_key_iv: "f106be47a4b0f7ea67aa309d7bcf8246"
server_key_version: "encryptionkey@0"
I20260927 21:37:53.607007 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "cd766495cb564398be0c38208ac7dbc6"
format_stamp: "Formatted at 2026-09-27 21:37:53 on dist-test-slave-pqr8"
server_key: "927d863a0165d312b10b8b6d4138d44a"
server_key_iv: "f106be47a4b0f7ea67aa309d7bcf8246"
server_key_version: "encryptionkey@0"
I20260927 21:37:53.611672 23097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:37:53.614777 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:53.615949 23097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:37:53.616115 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "cd766495cb564398be0c38208ac7dbc6"
format_stamp: "Formatted at 2026-09-27 21:37:53 on dist-test-slave-pqr8"
server_key: "927d863a0165d312b10b8b6d4138d44a"
server_key_iv: "f106be47a4b0f7ea67aa309d7bcf8246"
server_key_version: "encryptionkey@0"
I20260927 21:37:53.616302 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:53.632543 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:53.636080 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:53.636341 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:53.653630 23097 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:46063
I20260927 21:37:53.653632 23163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:46063 every 8 connection(s)
I20260927 21:37:53.654800 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:53.656621 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23097
I20260927 21:37:53.656847 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:37:53.657289 3819 external_mini_cluster.cc:1428] Setting key b857ac102b4ff9389b21a1476b12fe60
I20260927 21:37:53.658546 23164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:37:53 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:53.671681 23164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6: Bootstrap starting.
I20260927 21:37:53.674784 23164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:53.675732 23164 log.cc:824] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:53.678354 23164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6: No bootstrap required, opened a new log
I20260927 21:37:53.678884 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:53.658972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.3.186.254:46063)
0927 21:37:53.659684 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:53.659694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:53.659730 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:37:53.661199 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:53.661216 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:53.661599 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:53.661883 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:53.663601 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:53.664866 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:53.666725 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:53.667334 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:53.667477 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:53.672253 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:53.672260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:53.672271 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:53.672306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:53.675247 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:53.675905 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:53.675920 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:53.675926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:53.676080 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:53.676429 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:53.676434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:53.676439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:53.676880 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:53.677164 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:53.677624 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 21:37:53.677990 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":333,"threads_started":1}
I20260927 21:37:53.685766 23164 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd766495cb564398be0c38208ac7dbc6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46063 } }
I20260927 21:37:53.686113 23164 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:53.686215 23164 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd766495cb564398be0c38208ac7dbc6, State: Initialized, Role: FOLLOWER
I20260927 21:37:53.686841 23164 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd766495cb564398be0c38208ac7dbc6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46063 } }
I20260927 21:37:53.687039 23164 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:53.687143 23164 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:53.687254 23164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:53.689311 23164 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd766495cb564398be0c38208ac7dbc6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46063 } }
I20260927 21:37:53.689831 23164 leader_election.cc:302] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd766495cb564398be0c38208ac7dbc6; no voters:
I20260927 21:37:53.690311 23164 leader_election.cc:288] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:37:53.690459 23169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:53.691126 23169 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [term 1 LEADER]: Becoming Leader. State: Replica: cd766495cb564398be0c38208ac7dbc6, State: Running, Role: LEADER
I20260927 21:37:53.691630 23169 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd766495cb564398be0c38208ac7dbc6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46063 } }
I20260927 21:37:53.692312 23164 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:53.694240 23170 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd766495cb564398be0c38208ac7dbc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd766495cb564398be0c38208ac7dbc6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46063 } } }
I20260927 21:37:53.694550 23170 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:53.695007 23171 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd766495cb564398be0c38208ac7dbc6. Latest consensus state: current_term: 1 leader_uuid: "cd766495cb564398be0c38208ac7dbc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd766495cb564398be0c38208ac7dbc6" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 46063 } } }
I20260927 21:37:53.695358 23171 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:53.695359 23176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:53.700452 23176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:53.709040 23176 catalog_manager.cc:1347] Generated new cluster ID: 3db54cc420644de7aad841d7031d8aad
I20260927 21:37:53.709128 23176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:53.733143 23176 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:37:53.734344 23176 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:53.742993 23176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6: Generated new TSK 0
I20260927 21:37:53.743829 23176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
Entry for principal kudu/127.3.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab.
I20260927 21:37:53.800369 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46063
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:54.108476 23190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:54.108704 23190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:54.108801 23190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:54.115617 23190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:54.115705 23190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:54.115803 23190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:54.125206 23190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.193.keytab
--principal=kudu/127.3.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:54.126286 23190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:54.127403 23190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:54.139820 23195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:54.139776 23196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:54.140291 23198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:54.140574 23190 server_base.cc:1034] running on GCE node
I20260927 21:37:54.141088 23190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:54.141978 23190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:54.143203 23190 hybrid_clock.cc:648] HybridClock initialized: now 1790545074143160 us; error 49 us; skew 500 ppm
I20260927 21:37:54.143482 23190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:54 dist-test-slave-pqr8 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:54.146407 23190 init.cc:377] Logged in from keytab as kudu/127.3.186.193@KRBTEST.COM (short username kudu)
I20260927 21:37:54.147733 23190 webserver.cc:460] Webserver started at http://127.3.186.193:41175/ using document root <none> and password file <none>
I20260927 21:37:54.148396 23190 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:54.148548 23190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:54.148825 23190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:54.151347 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "e3b1784c493741efa5b6af7bd9e75293"
format_stamp: "Formatted at 2026-09-27 21:37:54 on dist-test-slave-pqr8"
server_key: "485c1edb1e26c5c423487dc9bfac8951"
server_key_iv: "48a6a23ffdb8b7c03c6dd15a3cac6d42"
server_key_version: "encryptionkey@0"
I20260927 21:37:54.152070 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "e3b1784c493741efa5b6af7bd9e75293"
format_stamp: "Formatted at 2026-09-27 21:37:54 on dist-test-slave-pqr8"
server_key: "485c1edb1e26c5c423487dc9bfac8951"
server_key_iv: "48a6a23ffdb8b7c03c6dd15a3cac6d42"
server_key_version: "encryptionkey@0"
I20260927 21:37:54.156661 23190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 21:37:54.161044 23205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:54.162253 23190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:37:54.162411 23190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "e3b1784c493741efa5b6af7bd9e75293"
format_stamp: "Formatted at 2026-09-27 21:37:54 on dist-test-slave-pqr8"
server_key: "485c1edb1e26c5c423487dc9bfac8951"
server_key_iv: "48a6a23ffdb8b7c03c6dd15a3cac6d42"
server_key_version: "encryptionkey@0"
I20260927 21:37:54.162603 23190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:54.174304 23190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:54.177655 23190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:54.177909 23190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:54.178799 23190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:54.180079 23190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:54.180172 23190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:54.180302 23190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:54.180373 23190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:54.213310 23190 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:40217
I20260927 21:37:54.213385 23317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:40217 every 8 connection(s)
I20260927 21:37:54.214437 23190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:54.219561 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23190
I20260927 21:37:54.219739 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:37:54.220281 3819 external_mini_cluster.cc:1428] Setting key 627634f1340cefee096257e39586a37b
Sep 27 21:37:54 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.193@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:54.235459 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:54.216613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:46139 (local address 127.3.186.254:46063)
0927 21:37:54.217007 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:54.217012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:54.217851 (+ 839us) server_negotiation.cc:407] Connection header received
0927 21:37:54.219866 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:54.219872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:54.219945 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:54.220037 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:54.221799 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.222674 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.225073 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.225445 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.225598 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:54.229400 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:54.229405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:54.229408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:54.229439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:54.232405 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:54.233254 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:54.233257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:54.233260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:54.233368 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:54.233835 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:54.233838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:54.233840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:54.234116 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:54.234224 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:54.235055 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 21:37:54.235201 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":214,"threads_started":1}
I20260927 21:37:54.236775 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:54.217015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46063 (local address 127.3.186.193:46139)
0927 21:37:54.217694 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:54.217739 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:54.219565 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:54.220303 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:54.220313 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:54.220974 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:54.221629 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:54.221653 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.222826 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.222832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:54.224933 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:54.224950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.225688 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.225697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:54.226021 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:54.226682 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:54.226714 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:54.229193 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:54.232552 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:54.232563 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:54.232578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:54.233142 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:54.233464 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:54.233471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:54.233478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:54.233725 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:54.234242 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:54.234264 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:54.234704 (+ 440us) client_negotiation.cc:769] Sending connection context
0927 21:37:54.234971 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 21:37:54.235468 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260927 21:37:54.238253 23318 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46063
I20260927 21:37:54.238546 23318 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:54.239162 23318 heartbeater.cc:511] Master 127.3.186.254:46063 requested a full tablet report, sending...
I20260927 21:37:54.241243 23129 ts_manager.cc:194] Registered new tserver with Master: e3b1784c493741efa5b6af7bd9e75293 (127.3.186.193:40217)
I20260927 21:37:54.243551 23129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.193@KRBTEST.COM'} at 127.3.186.193:46139
WARNING: no policy specified for kudu/127.3.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
Entry for principal kudu/127.3.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab.
I20260927 21:37:54.273002 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46063
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:54.572506 23324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:54.572736 23324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:54.572883 23324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:54.579605 23324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:54.579689 23324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:54.579890 23324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:54.589232 23324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.194.keytab
--principal=kudu/127.3.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:54.590348 23324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:54.591598 23324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:54.604663 23332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:54.604758 23329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:54.604712 23330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:54.605731 23324 server_base.cc:1034] running on GCE node
I20260927 21:37:54.606309 23324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:54.607115 23324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:54.608336 23324 hybrid_clock.cc:648] HybridClock initialized: now 1790545074608299 us; error 50 us; skew 500 ppm
I20260927 21:37:54.608644 23324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:54 dist-test-slave-pqr8 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:54.611708 23324 init.cc:377] Logged in from keytab as kudu/127.3.186.194@KRBTEST.COM (short username kudu)
I20260927 21:37:54.612998 23324 webserver.cc:460] Webserver started at http://127.3.186.194:40299/ using document root <none> and password file <none>
I20260927 21:37:54.613688 23324 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:54.613828 23324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:54.614130 23324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:54.616736 23324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "d7358afe18a54e43bca4d0adc48ccc22"
format_stamp: "Formatted at 2026-09-27 21:37:54 on dist-test-slave-pqr8"
server_key: "69ab2d729aaa48d46e57332fa088bdd4"
server_key_iv: "977dae05c62c802e3af1121b313d631f"
server_key_version: "encryptionkey@0"
I20260927 21:37:54.617559 23324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "d7358afe18a54e43bca4d0adc48ccc22"
format_stamp: "Formatted at 2026-09-27 21:37:54 on dist-test-slave-pqr8"
server_key: "69ab2d729aaa48d46e57332fa088bdd4"
server_key_iv: "977dae05c62c802e3af1121b313d631f"
server_key_version: "encryptionkey@0"
I20260927 21:37:54.622439 23324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:37:54.625723 23339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:54.626890 23324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 21:37:54.627066 23324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "d7358afe18a54e43bca4d0adc48ccc22"
format_stamp: "Formatted at 2026-09-27 21:37:54 on dist-test-slave-pqr8"
server_key: "69ab2d729aaa48d46e57332fa088bdd4"
server_key_iv: "977dae05c62c802e3af1121b313d631f"
server_key_version: "encryptionkey@0"
I20260927 21:37:54.627256 23324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:54.637377 23324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:54.641407 23324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:54.641705 23324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:54.642554 23324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:54.643713 23324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:54.643818 23324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:54.643968 23324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:54.644038 23324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:54.676957 23324 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:35163
I20260927 21:37:54.677032 23451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:35163 every 8 connection(s)
I20260927 21:37:54.678153 23324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:54.680513 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23324
I20260927 21:37:54.680694 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:37:54.681216 3819 external_mini_cluster.cc:1428] Setting key 43810758b08062fe447d19058aa297fe
Sep 27 21:37:54 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545074, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.194@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:54.698526 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:54.680475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:44549 (local address 127.3.186.254:46063)
0927 21:37:54.681690 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:54.681696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:54.682049 (+ 353us) server_negotiation.cc:407] Connection header received
0927 21:37:54.683945 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:54.683950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:54.684047 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:54.684163 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:54.685550 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.686275 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.687781 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.688199 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.688334 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:54.691800 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:54.691809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:54.691817 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:54.691880 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:54.695444 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:54.696108 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:54.696116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:54.696123 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:54.696285 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:54.696660 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:54.696668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:54.696673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:54.697036 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:54.697186 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:54.697823 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 21:37:54.698079 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1081}
I20260927 21:37:54.699054 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:54.681053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46063 (local address 127.3.186.194:44549)
0927 21:37:54.681862 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:54.681920 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:54.683695 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:54.684399 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:54.684407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:54.684878 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:54.685366 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:54.685382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.686438 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.686443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:54.687637 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:54.687648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:54.688427 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:54.688434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:54.688660 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:54.689434 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:54.689456 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:54.691606 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:54.695572 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:54.695577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:54.695586 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:54.695988 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:54.696390 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:54.696393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:54.696396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:54.696552 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:54.697148 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:54.697153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:54.697411 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 21:37:54.697768 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 21:37:54.698101 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":311,"threads_started":1}
I20260927 21:37:54.700376 23452 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46063
I20260927 21:37:54.700663 23452 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:54.701401 23452 heartbeater.cc:511] Master 127.3.186.254:46063 requested a full tablet report, sending...
I20260927 21:37:54.702904 23129 ts_manager.cc:194] Registered new tserver with Master: d7358afe18a54e43bca4d0adc48ccc22 (127.3.186.194:35163)
I20260927 21:37:54.703706 23129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.194@KRBTEST.COM'} at 127.3.186.194:44549
WARNING: no policy specified for kudu/127.3.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
Entry for principal kudu/127.3.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab.
I20260927 21:37:54.734784 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46063
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:37:55.034389 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:55.034660 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:55.034816 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:55.041718 23457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:55.041841 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:55.042003 23457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:55.051105 23457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.195.keytab
--principal=kudu/127.3.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:46063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:55.052217 23457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:55.053457 23457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:55.065932 23462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:55.065984 23463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:55.066020 23465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:55.067138 23457 server_base.cc:1034] running on GCE node
I20260927 21:37:55.067667 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:55.068547 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:55.069759 23457 hybrid_clock.cc:648] HybridClock initialized: now 1790545075069721 us; error 49 us; skew 500 ppm
I20260927 21:37:55.070061 23457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:37:55 dist-test-slave-pqr8 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545075, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:37:55.072924 23457 init.cc:377] Logged in from keytab as kudu/127.3.186.195@KRBTEST.COM (short username kudu)
I20260927 21:37:55.074258 23457 webserver.cc:460] Webserver started at http://127.3.186.195:44883/ using document root <none> and password file <none>
I20260927 21:37:55.074863 23457 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:55.075006 23457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:55.075289 23457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:55.077977 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c"
format_stamp: "Formatted at 2026-09-27 21:37:55 on dist-test-slave-pqr8"
server_key: "98b0bb9cfa6addee968a19f3a6a558f8"
server_key_iv: "8c8a14d43b4b53f258342d9b9087b1e4"
server_key_version: "encryptionkey@0"
I20260927 21:37:55.078714 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c"
format_stamp: "Formatted at 2026-09-27 21:37:55 on dist-test-slave-pqr8"
server_key: "98b0bb9cfa6addee968a19f3a6a558f8"
server_key_iv: "8c8a14d43b4b53f258342d9b9087b1e4"
server_key_version: "encryptionkey@0"
I20260927 21:37:55.083397 23457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:37:55.086550 23472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:55.087672 23457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:55.087836 23457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c"
format_stamp: "Formatted at 2026-09-27 21:37:55 on dist-test-slave-pqr8"
server_key: "98b0bb9cfa6addee968a19f3a6a558f8"
server_key_iv: "8c8a14d43b4b53f258342d9b9087b1e4"
server_key_version: "encryptionkey@0"
I20260927 21:37:55.088037 23457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:55.114391 23457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:55.117777 23457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:55.118049 23457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:55.118863 23457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:55.120085 23457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:55.120182 23457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:55.120352 23457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:55.120411 23457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:55.151988 23457 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:39021
I20260927 21:37:55.152072 23584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:39021 every 8 connection(s)
I20260927 21:37:55.153160 23457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:55.162779 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23457
I20260927 21:37:55.162987 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:37:55.163506 3819 external_mini_cluster.cc:1428] Setting key b29a91b6d040f7c4bca033d98c8f72d2
Sep 27 21:37:55 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545075, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.195@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:37:55.172272 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.155463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:38825 (local address 127.3.186.254:46063)
0927 21:37:55.155629 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.155633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.156693 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 21:37:55.158494 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.158498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.158560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.158646 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:55.159974 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.160677 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.162760 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.163449 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.163582 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.167504 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:55.167509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:55.167512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:55.167543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:55.169889 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.170540 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.170544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.170546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.170653 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.171028 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.171031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.171032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.171294 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:55.171401 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.171907 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.172042 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:37:55.173048 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.155915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:46063 (local address 127.3.186.195:38825)
0927 21:37:55.156540 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:55.156578 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:55.158258 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:55.158833 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:55.158841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:55.159317 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:55.159811 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.159827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.160830 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.160834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.162546 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.162564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.163879 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.163887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.164175 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.164828 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:37:55.164847 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:55.167327 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:55.170031 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:55.170037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:55.170047 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:55.170416 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:55.170738 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:37:55.170741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:37:55.170743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:37:55.170904 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:37:55.171387 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:55.171392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:55.171635 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 21:37:55.171819 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 21:37:55.172137 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260927 21:37:55.174214 23585 heartbeater.cc:347] Connected to a master server at 127.3.186.254:46063
I20260927 21:37:55.174489 23585 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:55.175133 23585 heartbeater.cc:511] Master 127.3.186.254:46063 requested a full tablet report, sending...
I20260927 21:37:55.176350 23129 ts_manager.cc:194] Registered new tserver with Master: 07e8f00ecf5b4f8a9558ed11a98bfe9c (127.3.186.195:39021)
I20260927 21:37:55.177034 23129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.186.195@KRBTEST.COM'} at 127.3.186.195:38825
I20260927 21:37:55.178727 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:55.195340 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.183034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.3.186.254:46063)
0927 21:37:55.183242 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.183250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.183721 (+ 471us) server_negotiation.cc:407] Connection header received
0927 21:37:55.183892 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.183896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.183963 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.184055 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:55.185412 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.186160 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.187419 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.187804 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.187929 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.190041 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:55.190046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:55.190051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:55.190087 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:55.192837 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.193548 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.193554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.193557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.193680 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.194044 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.194049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.194052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.194322 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:55.194570 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.194877 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.195062 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:37:55.198547 23128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:55.205184 23128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:55.237583 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.216871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.3.186.194:35163)
0927 21:37:55.226292 (+ 9421us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.226303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.226339 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:37:55.227302 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.227313 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.227590 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.227800 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:55.229637 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.231641 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.235002 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.236431 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.236598 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.236796 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.237069 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.237233 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9252,"thread_start_us":9159,"threads_started":1}
I20260927 21:37:55.237674 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.216358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:35163 (local address 127.0.0.1:60708)
0927 21:37:55.219444 (+ 3086us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:55.219484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:55.219621 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:55.228229 (+ 8608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:55.228240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:55.228280 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:55.228788 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.228802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.231854 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.231862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.234843 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.234857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.236642 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.236648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.236836 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.236882 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:37:55.237013 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 21:37:55.237192 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2875,"mutex_wait_us":2,"thread_start_us":2751,"threads_started":1}
I20260927 21:37:55.239023 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.217213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:39021 (local address 127.0.0.1:48220)
0927 21:37:55.229428 (+ 12215us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:55.229456 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:55.229848 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:55.231189 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:55.231196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:55.231229 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:55.231726 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.231759 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.234110 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.234117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.236467 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.236483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.238282 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.238288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.238475 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.238516 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:37:55.238579 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:37:55.238698 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12026,"mutex_wait_us":35,"thread_start_us":12316,"threads_started":1}
I20260927 21:37:55.240095 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.217477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.3.186.195:39021)
0927 21:37:55.230463 (+ 12986us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.230474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.230508 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:37:55.230624 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.230635 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.230957 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.231953 (+ 996us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:55.232212 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.233944 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.236653 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.238052 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.239395 (+ 1343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.239548 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.239633 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.239784 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12776,"thread_start_us":12595,"threads_started":1}
I20260927 21:37:55.242026 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.219807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:40217 (local address 127.0.0.1:34222)
0927 21:37:55.234590 (+ 14783us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:55.234606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:55.234722 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:55.235251 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:55.235255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:55.235279 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:55.235712 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.235721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.237206 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.237211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.239499 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.239512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.241263 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.241270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.241458 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.241488 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:37:55.241606 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 21:37:55.241721 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14638,"thread_start_us":14476,"threads_started":1}
I20260927 21:37:55.242112 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.220138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.3.186.193:40217)
0927 21:37:55.233917 (+ 13779us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.233925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.234713 (+ 788us) server_negotiation.cc:407] Connection header received
0927 21:37:55.234870 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.234877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.235087 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.235262 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:55.235877 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.237045 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.239656 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.241069 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.241220 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.241426 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.241642 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.241788 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13616,"thread_start_us":13778,"threads_started":1}
I20260927 21:37:55.241714 23387 tablet_service.cc:1467] Processing CreateTablet for tablet efdf6113ae42464aae1135e56ab9551f (DEFAULT_TABLE table=test-table [id=3dced0468a9c494c90d41f2d425c5c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:55.243140 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdf6113ae42464aae1135e56ab9551f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:55.243479 23520 tablet_service.cc:1467] Processing CreateTablet for tablet efdf6113ae42464aae1135e56ab9551f (DEFAULT_TABLE table=test-table [id=3dced0468a9c494c90d41f2d425c5c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:55.244746 23520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdf6113ae42464aae1135e56ab9551f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:55.244692 23253 tablet_service.cc:1467] Processing CreateTablet for tablet efdf6113ae42464aae1135e56ab9551f (DEFAULT_TABLE table=test-table [id=3dced0468a9c494c90d41f2d425c5c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:55.246114 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdf6113ae42464aae1135e56ab9551f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:55.246243 23318 heartbeater.cc:503] Master 127.3.186.254:46063 was elected leader, sending a full tablet report...
I20260927 21:37:55.256309 23603 tablet_bootstrap.cc:492] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Bootstrap starting.
I20260927 21:37:55.258215 23604 tablet_bootstrap.cc:492] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Bootstrap starting.
I20260927 21:37:55.259729 23605 tablet_bootstrap.cc:492] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Bootstrap starting.
I20260927 21:37:55.260032 23603 tablet_bootstrap.cc:654] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:55.261320 23603 log.cc:824] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:55.262580 23604 tablet_bootstrap.cc:654] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:55.262797 23605 tablet_bootstrap.cc:654] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:55.263562 23604 log.cc:824] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:55.263965 23605 log.cc:824] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:55.264349 23603 tablet_bootstrap.cc:492] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: No bootstrap required, opened a new log
I20260927 21:37:55.264667 23603 ts_tablet_manager.cc:1397] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 21:37:55.267475 23604 tablet_bootstrap.cc:492] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: No bootstrap required, opened a new log
I20260927 21:37:55.267715 23604 ts_tablet_manager.cc:1397] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 21:37:55.268096 23605 tablet_bootstrap.cc:492] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: No bootstrap required, opened a new log
I20260927 21:37:55.268332 23605 ts_tablet_manager.cc:1397] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 21:37:55.274086 23604 raft_consensus.cc:348] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.274533 23604 raft_consensus.cc:374] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:55.274650 23604 raft_consensus.cc:729] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7358afe18a54e43bca4d0adc48ccc22, State: Initialized, Role: FOLLOWER
I20260927 21:37:55.274668 23603 raft_consensus.cc:348] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.275031 23603 raft_consensus.cc:374] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:55.275116 23603 raft_consensus.cc:729] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b1784c493741efa5b6af7bd9e75293, State: Initialized, Role: FOLLOWER
I20260927 21:37:55.275276 23604 consensus_queue.cc:260] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.275645 23603 consensus_queue.cc:260] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.276436 23452 heartbeater.cc:503] Master 127.3.186.254:46063 was elected leader, sending a full tablet report...
I20260927 21:37:55.277045 23604 ts_tablet_manager.cc:1428] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260927 21:37:55.278961 23605 raft_consensus.cc:348] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.279345 23605 raft_consensus.cc:374] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:55.279454 23605 raft_consensus.cc:729] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e8f00ecf5b4f8a9558ed11a98bfe9c, State: Initialized, Role: FOLLOWER
I20260927 21:37:55.279853 23603 ts_tablet_manager.cc:1428] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 21:37:55.279978 23605 consensus_queue.cc:260] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.281286 23585 heartbeater.cc:503] Master 127.3.186.254:46063 was elected leader, sending a full tablet report...
I20260927 21:37:55.281890 23605 ts_tablet_manager.cc:1428] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260927 21:37:55.396220 23611 raft_consensus.cc:482] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:55.396503 23611 raft_consensus.cc:504] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.398236 23611 leader_election.cc:288] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7358afe18a54e43bca4d0adc48ccc22 (127.3.186.194:35163), e3b1784c493741efa5b6af7bd9e75293 (127.3.186.193:40217)
W20260927 21:37:55.404712 23586 tablet.cc:2369] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:55.404868 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.398645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:34833 (local address 127.3.186.194:35163)
0927 21:37:55.398874 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.398880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.399102 (+ 222us) server_negotiation.cc:407] Connection header received
0927 21:37:55.399191 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.399195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.399273 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.399378 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:55.399984 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.400981 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.402822 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.403943 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.404048 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.404096 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.404472 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.404600 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:37:55.404928 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.398673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:35163 (local address 127.3.186.195:34833)
0927 21:37:55.398881 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:55.398901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:55.399055 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:55.399572 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:55.399581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:55.399608 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:55.399800 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.399809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.401114 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.401120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.402646 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.402658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.404079 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.404086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.404270 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.404304 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:37:55.404418 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:37:55.404547 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 21:37:55.405942 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.399211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:47631 (local address 127.3.186.193:40217)
0927 21:37:55.399420 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.399426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.399867 (+ 441us) server_negotiation.cc:407] Connection header received
0927 21:37:55.400037 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.400041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.400095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.400160 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:55.400721 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.401713 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.403786 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.404921 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.405077 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.405140 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.405678 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.405765 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:37:55.405957 23407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdf6113ae42464aae1135e56ab9551f" candidate_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7358afe18a54e43bca4d0adc48ccc22" is_pre_election: true
I20260927 21:37:55.406183 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.399036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:40217 (local address 127.3.186.195:47631)
0927 21:37:55.399754 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:55.399771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:55.399874 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:55.400209 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:55.400216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:55.400243 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:55.400579 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.400590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.401853 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.401856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.403638 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:55.403653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.405301 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.405309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:55.405484 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.405544 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:37:55.405645 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:37:55.405776 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":1,"thread_start_us":388,"threads_started":1}
I20260927 21:37:55.406489 23407 raft_consensus.cc:2454] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e8f00ecf5b4f8a9558ed11a98bfe9c in term 0.
I20260927 21:37:55.407086 23475 leader_election.cc:302] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e8f00ecf5b4f8a9558ed11a98bfe9c, d7358afe18a54e43bca4d0adc48ccc22; no voters:
I20260927 21:37:55.407146 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdf6113ae42464aae1135e56ab9551f" candidate_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b1784c493741efa5b6af7bd9e75293" is_pre_election: true
I20260927 21:37:55.407454 23611 raft_consensus.cc:2781] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:55.407618 23611 raft_consensus.cc:482] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:55.407713 23611 raft_consensus.cc:3037] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:55.407704 23273 raft_consensus.cc:2454] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e8f00ecf5b4f8a9558ed11a98bfe9c in term 0.
I20260927 21:37:55.410089 23611 raft_consensus.cc:504] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.410735 23611 leader_election.cc:288] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [CANDIDATE]: Term 1 election: Requested vote from peers d7358afe18a54e43bca4d0adc48ccc22 (127.3.186.194:35163), e3b1784c493741efa5b6af7bd9e75293 (127.3.186.193:40217)
I20260927 21:37:55.411092 23407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdf6113ae42464aae1135e56ab9551f" candidate_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7358afe18a54e43bca4d0adc48ccc22"
I20260927 21:37:55.411170 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdf6113ae42464aae1135e56ab9551f" candidate_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b1784c493741efa5b6af7bd9e75293"
I20260927 21:37:55.411266 23407 raft_consensus.cc:3037] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:55.411391 23273 raft_consensus.cc:3037] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:55.413332 23407 raft_consensus.cc:2454] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e8f00ecf5b4f8a9558ed11a98bfe9c in term 1.
I20260927 21:37:55.413509 23273 raft_consensus.cc:2454] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e8f00ecf5b4f8a9558ed11a98bfe9c in term 1.
I20260927 21:37:55.413726 23475 leader_election.cc:302] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e8f00ecf5b4f8a9558ed11a98bfe9c, d7358afe18a54e43bca4d0adc48ccc22; no voters:
I20260927 21:37:55.414047 23611 raft_consensus.cc:2781] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:55.414443 23611 raft_consensus.cc:686] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 1 LEADER]: Becoming Leader. State: Replica: 07e8f00ecf5b4f8a9558ed11a98bfe9c, State: Running, Role: LEADER
I20260927 21:37:55.414880 23611 consensus_queue.cc:237] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } }
I20260927 21:37:55.419018 23128 catalog_manager.cc:5495] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 07e8f00ecf5b4f8a9558ed11a98bfe9c (127.3.186.195). New cstate: current_term: 1 leader_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07e8f00ecf5b4f8a9558ed11a98bfe9c" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 39021 } health_report { overall_health: HEALTHY } } }
W20260927 21:37:55.430035 23453 tablet.cc:2369] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:55.459686 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.455022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.3.186.195:39021)
0927 21:37:55.455182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.455186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.455201 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:37:55.455268 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.455272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.455334 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.455460 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:55.455772 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.456442 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.457841 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.458267 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.458372 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.458777 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:55.459211 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:55.459319 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.459440 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.459502 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 21:37:55.466768 23319 tablet.cc:2369] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:55.467233 23407 raft_consensus.cc:1261] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 1 FOLLOWER]: Refusing update from remote peer 07e8f00ecf5b4f8a9558ed11a98bfe9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:55.467336 23273 raft_consensus.cc:1261] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 1 FOLLOWER]: Refusing update from remote peer 07e8f00ecf5b4f8a9558ed11a98bfe9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:55.468125 23613 consensus_queue.cc:1035] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7358afe18a54e43bca4d0adc48ccc22" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:55.468456 23611 consensus_queue.cc:1035] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b1784c493741efa5b6af7bd9e75293" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:55.479938 23616 mvcc.cc:205] Tried to move back new op lower bound from 7334072629104779264 to 7334072628902912000. Current Snapshot: MvccSnapshot[applied={T|T < 7334072629104779264}]
I20260927 21:37:55.491189 23129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:55.491452 23129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:55.495031 23129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd766495cb564398be0c38208ac7dbc6: Sending DeleteTablet for 3 replicas of tablet efdf6113ae42464aae1135e56ab9551f
I20260927 21:37:55.496133 23520 tablet_service.cc:1514] Processing DeleteTablet for tablet efdf6113ae42464aae1135e56ab9551f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:55 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:48220
I20260927 21:37:55.496133 23387 tablet_service.cc:1514] Processing DeleteTablet for tablet efdf6113ae42464aae1135e56ab9551f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:55 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:60708
I20260927 21:37:55.496331 23253 tablet_service.cc:1514] Processing DeleteTablet for tablet efdf6113ae42464aae1135e56ab9551f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:55 UTC) from {username='kudu', principal='kudu/127.3.186.254@KRBTEST.COM'} at 127.0.0.1:34222
I20260927 21:37:55.497087 23627 tablet_replica.cc:330] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: stopping tablet replica
I20260927 21:37:55.497462 23627 raft_consensus.cc:2229] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:55.497849 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23190
I20260927 21:37:55.498026 23627 raft_consensus.cc:2258] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:55.500312 23627 ts_tablet_manager.cc:1905] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:55.507700 23627 ts_tablet_manager.cc:1918] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:55.507920 23627 log.cc:1196] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/efdf6113ae42464aae1135e56ab9551f
I20260927 21:37:55.508569 23627 ts_tablet_manager.cc:1939] T efdf6113ae42464aae1135e56ab9551f P e3b1784c493741efa5b6af7bd9e75293: Deleting consensus metadata
I20260927 21:37:55.510370 23115 catalog_manager.cc:4841] TS e3b1784c493741efa5b6af7bd9e75293 (127.3.186.193:40217): tablet efdf6113ae42464aae1135e56ab9551f (table test-table [id=3dced0468a9c494c90d41f2d425c5c58]) successfully deleted
I20260927 21:37:55.514603 23625 tablet_replica.cc:330] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: stopping tablet replica
I20260927 21:37:55.515163 23626 tablet_replica.cc:330] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: stopping tablet replica
I20260927 21:37:55.515560 23626 raft_consensus.cc:2229] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:55.515643 23625 raft_consensus.cc:2229] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:55.516161 23625 raft_consensus.cc:2258] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:55.516146 23626 raft_consensus.cc:2258] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:55.518083 23625 ts_tablet_manager.cc:1905] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:55.518146 23626 ts_tablet_manager.cc:1905] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:37:55 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.193@KRBTEST.COM
I20260927 21:37:55.525251 23625 ts_tablet_manager.cc:1918] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:55.525465 23625 log.cc:1196] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/efdf6113ae42464aae1135e56ab9551f
I20260927 21:37:55.526101 23625 ts_tablet_manager.cc:1939] T efdf6113ae42464aae1135e56ab9551f P d7358afe18a54e43bca4d0adc48ccc22: Deleting consensus metadata
I20260927 21:37:55.526203 23626 ts_tablet_manager.cc:1918] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:55.526367 23626 log.cc:1196] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/efdf6113ae42464aae1135e56ab9551f
I20260927 21:37:55.526939 23626 ts_tablet_manager.cc:1939] T efdf6113ae42464aae1135e56ab9551f P 07e8f00ecf5b4f8a9558ed11a98bfe9c: Deleting consensus metadata
I20260927 21:37:55.528507 23115 catalog_manager.cc:4841] TS d7358afe18a54e43bca4d0adc48ccc22 (127.3.186.194:35163): tablet efdf6113ae42464aae1135e56ab9551f (table test-table [id=3dced0468a9c494c90d41f2d425c5c58]) successfully deleted
I20260927 21:37:55.528561 23114 catalog_manager.cc:4841] TS 07e8f00ecf5b4f8a9558ed11a98bfe9c (127.3.186.195:39021): tablet efdf6113ae42464aae1135e56ab9551f (table test-table [id=3dced0468a9c494c90d41f2d425c5c58]) successfully deleted
I20260927 21:37:55.532164 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.499950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.3.186.193:40217)
0927 21:37:55.500148 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.500155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.513998 (+ 13843us) server_negotiation.cc:407] Connection header received
0927 21:37:55.514180 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.514188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.514282 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.514408 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:55.516135 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.517204 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.519116 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.519712 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.519799 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.525550 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:55.525564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:55.525578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:55.525624 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:55.529186 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.530099 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.530106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.530108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.530223 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.530672 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.530679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.530683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.531014 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:55.531233 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.531696 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.531856 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:37:55.532841 23313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:55.621029 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23190
I20260927 21:37:55.639389 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23324
Sep 27 21:37:55 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.194@KRBTEST.COM
I20260927 21:37:55.656353 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.640222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.3.186.194:35163)
0927 21:37:55.640418 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.640424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.640445 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:37:55.640589 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.640595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.640683 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.640791 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:55.642251 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.643359 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.644554 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.645156 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.645277 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.649426 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:55.649438 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:55.649458 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:55.649494 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:55.652999 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.654129 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.654139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.654142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.654280 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.654651 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.654656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.654660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.655118 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:55.655406 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.655803 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.656007 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:37:55.656967 23447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:55.758884 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23324
I20260927 21:37:55.775393 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23457
Sep 27 21:37:55 dist-test-slave-pqr8 krb5kdc[23084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.195@KRBTEST.COM
I20260927 21:37:55.800763 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:55.776210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.3.186.195:39021)
0927 21:37:55.776384 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:55.776391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:55.776603 (+ 212us) server_negotiation.cc:407] Connection header received
0927 21:37:55.776788 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:55.776792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:55.776857 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:55.776936 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:55.778733 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.779560 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.781470 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:55.782024 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:55.782162 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:55.795004 (+ 12842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:55.795014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:55.795027 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:37:55.795069 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:55.798101 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.798800 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.798813 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.798816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.798958 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:37:55.799326 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:37:55.799331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:37:55.799335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:37:55.799671 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:55.800008 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:55.800233 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:37:55.800443 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:37:55.801458 23580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:55.917608 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23457
I20260927 21:37:55.936797 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23097
I20260927 21:37:55.937858 23159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:56.042953 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23097
2026-09-27T21:37:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4064 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T21:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:37:56Z Disabled control of system clock
I20260927 21:37:56.112778 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43195
--webserver_interface=127.3.186.254
--webserver_port=0
--builtin_ntp_servers=127.3.186.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:43195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:37:56.416288 23636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:56.416527 23636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:56.416599 23636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:56.423318 23636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:37:56.423411 23636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:56.423456 23636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:56.423511 23636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:37:56.423552 23636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:37:56.432027 23636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:43195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:56.433035 23636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:56.434587 23636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:56.445989 23641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:56.445946 23644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:56.446028 23642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:56.446805 23636 server_base.cc:1034] running on GCE node
I20260927 21:37:56.447448 23636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:56.448642 23636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:56.449846 23636 hybrid_clock.cc:648] HybridClock initialized: now 1790545076449813 us; error 30 us; skew 500 ppm
I20260927 21:37:56.450147 23636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:56.452158 23636 webserver.cc:460] Webserver started at http://127.3.186.254:33273/ using document root <none> and password file <none>
I20260927 21:37:56.452711 23636 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:56.452790 23636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:56.453023 23636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:56.455602 23636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "d180b74f7f5442118dff951aad7a6e00"
format_stamp: "Formatted at 2026-09-27 21:37:56 on dist-test-slave-pqr8"
server_key: "374a8823bb37f773ae49c4c6be702d72"
server_key_iv: "14bd3d8816622d718bdd65b899cf7ddb"
server_key_version: "encryptionkey@0"
I20260927 21:37:56.456262 23636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "d180b74f7f5442118dff951aad7a6e00"
format_stamp: "Formatted at 2026-09-27 21:37:56 on dist-test-slave-pqr8"
server_key: "374a8823bb37f773ae49c4c6be702d72"
server_key_iv: "14bd3d8816622d718bdd65b899cf7ddb"
server_key_version: "encryptionkey@0"
I20260927 21:37:56.460942 23636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 21:37:56.463984 23650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:56.465209 23636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:37:56.465409 23636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "d180b74f7f5442118dff951aad7a6e00"
format_stamp: "Formatted at 2026-09-27 21:37:56 on dist-test-slave-pqr8"
server_key: "374a8823bb37f773ae49c4c6be702d72"
server_key_iv: "14bd3d8816622d718bdd65b899cf7ddb"
server_key_version: "encryptionkey@0"
I20260927 21:37:56.465654 23636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:56.482667 23636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:56.483335 23636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:56.483616 23636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:56.501715 23636 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:43195
I20260927 21:37:56.501739 23701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:43195 every 8 connection(s)
I20260927 21:37:56.502903 23636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:37:56.506088 23702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:56.509758 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23636
I20260927 21:37:56.509928 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:37:56.510414 3819 external_mini_cluster.cc:1428] Setting key 1d60a209911ddd598463eeec945a0758
I20260927 21:37:56.516641 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00: Bootstrap starting.
I20260927 21:37:56.519961 23702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:56.521073 23702 log.cc:824] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:56.523945 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00: No bootstrap required, opened a new log
I20260927 21:37:56.524233 23705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:56.512134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.3.186.254:43195)
0927 21:37:56.513035 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:56.513045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:56.513082 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:37:56.515339 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:56.515359 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:56.515744 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:56.516033 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:56.516648 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:56.517985 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:56.519806 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:56.520540 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:56.520670 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:56.521987 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:56.521998 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:56.522015 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:56.522651 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:56.522725 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:56.522744 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:56.522828 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:56.523059 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 21:37:56.523371 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":362,"threads_started":1}
I20260927 21:37:56.531294 23702 raft_consensus.cc:348] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d180b74f7f5442118dff951aad7a6e00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43195 } }
I20260927 21:37:56.531627 23702 raft_consensus.cc:374] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:56.531728 23702 raft_consensus.cc:729] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d180b74f7f5442118dff951aad7a6e00, State: Initialized, Role: FOLLOWER
I20260927 21:37:56.532227 23702 consensus_queue.cc:260] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d180b74f7f5442118dff951aad7a6e00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43195 } }
I20260927 21:37:56.532467 23702 raft_consensus.cc:388] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:37:56.532605 23702 raft_consensus.cc:482] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:37:56.532766 23702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:56.534953 23702 raft_consensus.cc:504] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d180b74f7f5442118dff951aad7a6e00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43195 } }
I20260927 21:37:56.535403 23702 leader_election.cc:302] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d180b74f7f5442118dff951aad7a6e00; no voters:
I20260927 21:37:56.535905 23702 leader_election.cc:288] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:37:56.536053 23707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:56.536732 23707 raft_consensus.cc:686] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [term 1 LEADER]: Becoming Leader. State: Replica: d180b74f7f5442118dff951aad7a6e00, State: Running, Role: LEADER
I20260927 21:37:56.537261 23707 consensus_queue.cc:237] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d180b74f7f5442118dff951aad7a6e00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43195 } }
I20260927 21:37:56.537828 23702 sys_catalog.cc:564] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:37:56.539924 23708 sys_catalog.cc:455] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d180b74f7f5442118dff951aad7a6e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d180b74f7f5442118dff951aad7a6e00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43195 } } }
I20260927 21:37:56.539901 23709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d180b74f7f5442118dff951aad7a6e00. Latest consensus state: current_term: 1 leader_uuid: "d180b74f7f5442118dff951aad7a6e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d180b74f7f5442118dff951aad7a6e00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43195 } } }
I20260927 21:37:56.540334 23708 sys_catalog.cc:458] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:56.540333 23709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00 [sys.catalog]: This master's current role is: LEADER
I20260927 21:37:56.542868 23713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:37:56.547300 23713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:37:56.555321 23713 catalog_manager.cc:1347] Generated new cluster ID: 1654c67d940b4449ba23032fa1ce7d9b
I20260927 21:37:56.555410 23713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:37:56.592339 23713 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:37:56.593602 23713 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:37:56.599778 23713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00: Generated new TSK 0
I20260927 21:37:56.600509 23713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:37:56.616473 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.193:0
--local_ip_for_outbound_sockets=127.3.186.193
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43195
--builtin_ntp_servers=127.3.186.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:37:56.932220 23726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:56.932451 23726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:56.932526 23726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:56.939394 23726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:56.939479 23726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:56.939570 23726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.193
I20260927 21:37:56.948233 23726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.186.193
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.193
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:56.949213 23726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:56.950762 23726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:56.962829 23731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:56.962781 23732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:56.962872 23734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:56.963816 23726 server_base.cc:1034] running on GCE node
I20260927 21:37:56.964375 23726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:56.965173 23726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:56.966351 23726 hybrid_clock.cc:648] HybridClock initialized: now 1790545076966284 us; error 58 us; skew 500 ppm
I20260927 21:37:56.966761 23726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:56.969048 23726 webserver.cc:460] Webserver started at http://127.3.186.193:44205/ using document root <none> and password file <none>
I20260927 21:37:56.969758 23726 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:56.969877 23726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:56.970170 23726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:56.972663 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data/instance:
uuid: "d4f91016e8d741eaa4468907fd62cceb"
format_stamp: "Formatted at 2026-09-27 21:37:56 on dist-test-slave-pqr8"
server_key: "b35e59e8a190a27625747c5cf58331a5"
server_key_iv: "f951b4e3ed0477e1e3ff5a8904a8fadb"
server_key_version: "encryptionkey@0"
I20260927 21:37:56.973389 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance:
uuid: "d4f91016e8d741eaa4468907fd62cceb"
format_stamp: "Formatted at 2026-09-27 21:37:56 on dist-test-slave-pqr8"
server_key: "b35e59e8a190a27625747c5cf58331a5"
server_key_iv: "f951b4e3ed0477e1e3ff5a8904a8fadb"
server_key_version: "encryptionkey@0"
I20260927 21:37:56.977902 23726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:37:56.981011 23740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:56.982178 23726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:56.982362 23726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal
uuid: "d4f91016e8d741eaa4468907fd62cceb"
format_stamp: "Formatted at 2026-09-27 21:37:56 on dist-test-slave-pqr8"
server_key: "b35e59e8a190a27625747c5cf58331a5"
server_key_iv: "f951b4e3ed0477e1e3ff5a8904a8fadb"
server_key_version: "encryptionkey@0"
I20260927 21:37:56.982594 23726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:56.993825 23726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:56.994506 23726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:56.994782 23726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:56.995616 23726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:56.996863 23726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:56.996980 23726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:56.997113 23726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:56.997216 23726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.028469 23726 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.193:33991
I20260927 21:37:57.028543 23852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.193:33991 every 8 connection(s)
I20260927 21:37:57.029670 23726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/data/info.pb
I20260927 21:37:57.037577 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23726
I20260927 21:37:57.037750 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal/instance
I20260927 21:37:57.038197 3819 external_mini_cluster.cc:1428] Setting key 997473c28bba885c0f5e5676dfa91b8f
I20260927 21:37:57.041085 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.194:0
--local_ip_for_outbound_sockets=127.3.186.194
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43195
--builtin_ntp_servers=127.3.186.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:37:57.042055 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.031887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:56169 (local address 127.3.186.254:43195)
0927 21:37:57.032293 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.032298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.033327 (+ 1029us) server_negotiation.cc:407] Connection header received
0927 21:37:57.035032 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.035037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.035124 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.035248 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:57.036231 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.036955 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.038909 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.039483 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.039592 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.041015 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:57.041023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:57.041030 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:57.041050 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:57.041104 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:57.041120 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:57.041218 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.041486 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.041617 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":206,"threads_started":1}
I20260927 21:37:57.042699 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.032398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43195 (local address 127.3.186.193:56169)
0927 21:37:57.033187 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:57.033223 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:57.034777 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:57.035467 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:57.035475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:57.035557 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:57.036059 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.036075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.037104 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.037110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.038747 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.038763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.039705 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.039712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.040012 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.040690 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:57.040693 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:57.040729 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:57.040793 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:57.041225 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:57.041239 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:57.041249 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:37:57.041457 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 21:37:57.041767 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":262,"threads_started":1}
I20260927 21:37:57.046870 23853 heartbeater.cc:347] Connected to a master server at 127.3.186.254:43195
I20260927 21:37:57.047194 23853 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:57.047971 23853 heartbeater.cc:511] Master 127.3.186.254:43195 requested a full tablet report, sending...
I20260927 21:37:57.050105 23667 ts_manager.cc:194] Registered new tserver with Master: d4f91016e8d741eaa4468907fd62cceb (127.3.186.193:33991)
I20260927 21:37:57.051540 23667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.193:56169
W20260927 21:37:57.352051 23857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:57.352330 23857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:57.352452 23857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:57.359247 23857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:57.359328 23857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:57.359499 23857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.194
I20260927 21:37:57.367810 23857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.186.194
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.194
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:57.368932 23857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:57.370603 23857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:57.383138 23865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:57.383177 23863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:57.383369 23862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:57.386907 23857 server_base.cc:1034] running on GCE node
I20260927 21:37:57.387454 23857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:57.388309 23857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:57.389582 23857 hybrid_clock.cc:648] HybridClock initialized: now 1790545077389530 us; error 56 us; skew 500 ppm
I20260927 21:37:57.389879 23857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:57.391944 23857 webserver.cc:460] Webserver started at http://127.3.186.194:38883/ using document root <none> and password file <none>
I20260927 21:37:57.392575 23857 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:57.392701 23857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:57.392993 23857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:57.395641 23857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data/instance:
uuid: "f4116a32f1b74451b4812bce2b437047"
format_stamp: "Formatted at 2026-09-27 21:37:57 on dist-test-slave-pqr8"
server_key: "9d26d7e3d33d79185a240c2a4f1cc231"
server_key_iv: "af17f36a96b067d56a21874df34b21e0"
server_key_version: "encryptionkey@0"
I20260927 21:37:57.396359 23857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance:
uuid: "f4116a32f1b74451b4812bce2b437047"
format_stamp: "Formatted at 2026-09-27 21:37:57 on dist-test-slave-pqr8"
server_key: "9d26d7e3d33d79185a240c2a4f1cc231"
server_key_iv: "af17f36a96b067d56a21874df34b21e0"
server_key_version: "encryptionkey@0"
I20260927 21:37:57.400985 23857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 21:37:57.404379 23871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.405668 23857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:37:57.405824 23857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal
uuid: "f4116a32f1b74451b4812bce2b437047"
format_stamp: "Formatted at 2026-09-27 21:37:57 on dist-test-slave-pqr8"
server_key: "9d26d7e3d33d79185a240c2a4f1cc231"
server_key_iv: "af17f36a96b067d56a21874df34b21e0"
server_key_version: "encryptionkey@0"
I20260927 21:37:57.406037 23857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:57.418740 23857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:57.419404 23857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:57.419677 23857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:57.420509 23857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:57.421856 23857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:57.421972 23857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.422107 23857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:57.422194 23857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.453151 23857 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.194:36513
I20260927 21:37:57.453219 23983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.194:36513 every 8 connection(s)
I20260927 21:37:57.454310 23857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/data/info.pb
I20260927 21:37:57.464352 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23857
I20260927 21:37:57.464541 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal/instance
I20260927 21:37:57.464850 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.456544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.194:49921 (local address 127.3.186.254:43195)
0927 21:37:57.456723 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.456728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.457939 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 21:37:57.459575 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.459579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.459639 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.459719 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:57.460600 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.461222 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.462594 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.462942 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.463025 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.464123 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:57.464127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:57.464130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:57.464146 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:57.464173 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:57.464184 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:57.464247 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.464498 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.464595 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:37:57.465054 3819 external_mini_cluster.cc:1428] Setting key b70cfdc9f9175332700e26006536e81b
I20260927 21:37:57.465890 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.456969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43195 (local address 127.3.186.194:49921)
0927 21:37:57.457820 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:57.457853 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:57.459354 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:57.459917 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:57.459923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:57.460012 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:57.460472 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.460486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.461339 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.461342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.462481 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.462492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.463098 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.463101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.463281 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.463832 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:57.463835 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:57.463868 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:57.463930 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:57.464278 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:57.464288 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:57.464295 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:37:57.464477 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 21:37:57.464807 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":245,"threads_started":1}
I20260927 21:37:57.467176 23984 heartbeater.cc:347] Connected to a master server at 127.3.186.254:43195
I20260927 21:37:57.467437 23984 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:57.467801 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.186.195:0
--local_ip_for_outbound_sockets=127.3.186.195
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43195
--builtin_ntp_servers=127.3.186.212:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:37:57.468173 23984 heartbeater.cc:511] Master 127.3.186.254:43195 requested a full tablet report, sending...
I20260927 21:37:57.469414 23666 ts_manager.cc:194] Registered new tserver with Master: f4116a32f1b74451b4812bce2b437047 (127.3.186.194:36513)
I20260927 21:37:57.470403 23666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.194:49921
W20260927 21:37:57.772223 23987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:37:57.772452 23987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:37:57.772578 23987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:37:57.779373 23987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:37:57.779454 23987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:37:57.779635 23987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.186.195
I20260927 21:37:57.791580 23987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.186.195
--webserver_port=0
--tserver_master_addrs=127.3.186.254:43195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.186.195
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:37:57.792853 23987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:37:57.795022 23987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:37:57.808323 23993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:57.808444 23995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:37:57.808409 23992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:37:57.809242 23987 server_base.cc:1034] running on GCE node
I20260927 21:37:57.809865 23987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:37:57.810765 23987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:37:57.811973 23987 hybrid_clock.cc:648] HybridClock initialized: now 1790545077811921 us; error 42 us; skew 500 ppm
I20260927 21:37:57.812261 23987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:37:57.814466 23987 webserver.cc:460] Webserver started at http://127.3.186.195:41339/ using document root <none> and password file <none>
I20260927 21:37:57.815088 23987 fs_manager.cc:359] Metadata directory not provided
I20260927 21:37:57.815205 23987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:37:57.815490 23987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:37:57.818121 23987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data/instance:
uuid: "1230b5e0eaba4b59b200aca1d582de04"
format_stamp: "Formatted at 2026-09-27 21:37:57 on dist-test-slave-pqr8"
server_key: "0cea8cf530083ce242258f2f941b6118"
server_key_iv: "a578eed08f1651ac16d589c51b6523dd"
server_key_version: "encryptionkey@0"
I20260927 21:37:57.818866 23987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance:
uuid: "1230b5e0eaba4b59b200aca1d582de04"
format_stamp: "Formatted at 2026-09-27 21:37:57 on dist-test-slave-pqr8"
server_key: "0cea8cf530083ce242258f2f941b6118"
server_key_iv: "a578eed08f1651ac16d589c51b6523dd"
server_key_version: "encryptionkey@0"
I20260927 21:37:57.823464 23987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:37:57.826574 24001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.827694 23987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:37:57.827852 23987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal
uuid: "1230b5e0eaba4b59b200aca1d582de04"
format_stamp: "Formatted at 2026-09-27 21:37:57 on dist-test-slave-pqr8"
server_key: "0cea8cf530083ce242258f2f941b6118"
server_key_iv: "a578eed08f1651ac16d589c51b6523dd"
server_key_version: "encryptionkey@0"
I20260927 21:37:57.828070 23987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:37:57.839937 23987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:37:57.840569 23987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:37:57.840830 23987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:37:57.841741 23987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:37:57.843103 23987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:37:57.843211 23987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.843328 23987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:37:57.843407 23987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:37:57.874620 23987 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.195:46013
I20260927 21:37:57.874691 24113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.195:46013 every 8 connection(s)
I20260927 21:37:57.875769 23987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/data/info.pb
I20260927 21:37:57.881847 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 23987
I20260927 21:37:57.882000 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal/instance
I20260927 21:37:57.882434 3819 external_mini_cluster.cc:1428] Setting key 26c0a6df1a2216c8680fa505be314b32
I20260927 21:37:57.887591 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.877966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.195:42667 (local address 127.3.186.254:43195)
0927 21:37:57.878151 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.878156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.879376 (+ 1220us) server_negotiation.cc:407] Connection header received
0927 21:37:57.880963 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.880967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.881032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.881116 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:57.882222 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.883061 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.885188 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.885702 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.885805 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.886993 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:57.886998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:57.887001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:57.887016 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:57.887046 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:57.887056 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:57.887124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.887281 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.887346 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:37:57.888465 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.878378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43195 (local address 127.3.186.195:42667)
0927 21:37:57.879211 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:57.879249 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:57.880729 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:57.881292 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:57.881299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:57.881375 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:37:57.882005 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.882022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.883207 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.883211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.885035 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.885048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.885900 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.885905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.886098 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.886710 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:37:57.886713 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:37:57.886745 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:37:57.886845 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:37:57.887137 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:37:57.887144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:37:57.887149 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:37:57.887277 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:37:57.887518 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":290,"threads_started":1}
I20260927 21:37:57.889742 24114 heartbeater.cc:347] Connected to a master server at 127.3.186.254:43195
I20260927 21:37:57.890008 24114 heartbeater.cc:464] Registering TS with master...
I20260927 21:37:57.890619 24114 heartbeater.cc:511] Master 127.3.186.254:43195 requested a full tablet report, sending...
I20260927 21:37:57.891774 23666 ts_manager.cc:194] Registered new tserver with Master: 1230b5e0eaba4b59b200aca1d582de04 (127.3.186.195:46013)
I20260927 21:37:57.892488 23666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.186.195:42667
I20260927 21:37:57.896958 3819 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:37:57.905730 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.900750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.3.186.254:43195)
0927 21:37:57.900981 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.900987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.901230 (+ 243us) server_negotiation.cc:407] Connection header received
0927 21:37:57.901346 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.901350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.901416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.901486 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:57.901957 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.902565 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.903897 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.904252 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.904341 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.905181 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:57.905186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:57.905188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:57.905204 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:57.905231 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:57.905242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:57.905307 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.905416 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.905477 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:37:57.909108 23666 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:37:57.915774 23666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:37:57.940604 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.926877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.3.186.195:46013)
0927 21:37:57.927720 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.927730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.928697 (+ 967us) server_negotiation.cc:407] Connection header received
0927 21:37:57.929107 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.929117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.929424 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.929642 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:57.931317 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.933340 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.936857 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.938640 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.938782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.939103 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.939972 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.940181 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":513,"threads_started":1}
I20260927 21:37:57.941493 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.926443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:46013 (local address 127.0.0.1:38170)
0927 21:37:57.928516 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:57.928573 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:57.928856 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:57.930434 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:57.930444 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:57.930485 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:57.931070 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.931086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.933628 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.933636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.936593 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.936612 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.939351 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.939359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.939670 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.939735 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 21:37:57.940823 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0927 21:37:57.941046 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1759,"thread_start_us":13940,"threads_started":1}
I20260927 21:37:57.945104 24049 tablet_service.cc:1467] Processing CreateTablet for tablet ea0add7d6f8f47d186691e855d8821e4 (DEFAULT_TABLE table=test-table [id=168401ca5d0243319e23d92992e23d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:57.946580 24049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0add7d6f8f47d186691e855d8821e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:57.950085 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.928856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:36513 (local address 127.0.0.1:45762)
0927 21:37:57.942080 (+ 13224us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:57.942096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:57.942210 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:57.942769 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:57.942773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:57.942798 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:57.943061 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.943070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.944828 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.944832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.947526 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.947539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.949352 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.949356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.949452 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.949472 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 21:37:57.949619 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 21:37:57.949742 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13121,"mutex_wait_us":108,"thread_start_us":13000,"threads_started":1}
I20260927 21:37:57.950152 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.929551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.3.186.194:36513)
0927 21:37:57.937877 (+ 8326us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.937893 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.942171 (+ 4278us) server_negotiation.cc:407] Connection header received
0927 21:37:57.942357 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.942365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.942560 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.942746 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:57.943221 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.944510 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.947715 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.948923 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.949059 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.949260 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.949667 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.949815 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8172,"thread_start_us":8049,"threads_started":1}
I20260927 21:37:57.950888 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.927964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.3.186.193:33991)
0927 21:37:57.943127 (+ 15163us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:57.943132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:57.943181 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:37:57.943273 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:57.943279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:57.943475 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:57.943649 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:57.944156 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.945487 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.947784 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.949498 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.949654 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.949850 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:57.950412 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 21:37:57.950563 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14543,"thread_start_us":14932,"threads_started":1}
I20260927 21:37:57.951556 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:57.927688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45518)
0927 21:37:57.940666 (+ 12978us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:57.940684 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:57.940796 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:57.943661 (+ 2865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:57.943665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:57.943686 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:57.943991 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.944003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.945685 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.945693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.947580 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:57.947597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:57.949979 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:57.949985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:57.950139 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:57.950166 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 21:37:57.951113 (+ 947us) client_negotiation.cc:240] Negotiation successful
0927 21:37:57.951231 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12846,"mutex_wait_us":38,"thread_start_us":336,"threads_started":1}
I20260927 21:37:57.953409 23919 tablet_service.cc:1467] Processing CreateTablet for tablet ea0add7d6f8f47d186691e855d8821e4 (DEFAULT_TABLE table=test-table [id=168401ca5d0243319e23d92992e23d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:57.954563 23919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0add7d6f8f47d186691e855d8821e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:57.954712 23788 tablet_service.cc:1467] Processing CreateTablet for tablet ea0add7d6f8f47d186691e855d8821e4 (DEFAULT_TABLE table=test-table [id=168401ca5d0243319e23d92992e23d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:37:57.955909 23788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0add7d6f8f47d186691e855d8821e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:37:57.960541 24132 tablet_bootstrap.cc:492] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Bootstrap starting.
I20260927 21:37:57.964071 24132 tablet_bootstrap.cc:654] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:57.965226 24132 log.cc:824] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:57.967983 24132 tablet_bootstrap.cc:492] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: No bootstrap required, opened a new log
I20260927 21:37:57.968240 24132 ts_tablet_manager.cc:1397] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:37:57.968407 24134 tablet_bootstrap.cc:492] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Bootstrap starting.
I20260927 21:37:57.972774 24134 tablet_bootstrap.cc:654] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:57.974107 24134 log.cc:824] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:57.974156 24135 tablet_bootstrap.cc:492] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Bootstrap starting.
I20260927 21:37:57.976400 24132 raft_consensus.cc:348] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:57.976737 24134 tablet_bootstrap.cc:492] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: No bootstrap required, opened a new log
I20260927 21:37:57.976822 24132 raft_consensus.cc:374] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:57.976922 24132 raft_consensus.cc:729] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1230b5e0eaba4b59b200aca1d582de04, State: Initialized, Role: FOLLOWER
I20260927 21:37:57.977021 24134 ts_tablet_manager.cc:1397] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 21:37:57.977480 24135 tablet_bootstrap.cc:654] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Neither blocks nor log segments found. Creating new log.
I20260927 21:37:57.977530 24132 consensus_queue.cc:260] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:57.979189 24132 ts_tablet_manager.cc:1428] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 21:37:57.979264 24114 heartbeater.cc:503] Master 127.3.186.254:43195 was elected leader, sending a full tablet report...
I20260927 21:37:57.979347 24135 log.cc:824] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:37:57.982107 24135 tablet_bootstrap.cc:492] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: No bootstrap required, opened a new log
I20260927 21:37:57.982347 24135 ts_tablet_manager.cc:1397] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 21:37:57.985184 24134 raft_consensus.cc:348] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:57.985538 24134 raft_consensus.cc:374] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:57.985652 24134 raft_consensus.cc:729] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4116a32f1b74451b4812bce2b437047, State: Initialized, Role: FOLLOWER
I20260927 21:37:57.986217 24134 consensus_queue.cc:260] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:57.987205 23984 heartbeater.cc:503] Master 127.3.186.254:43195 was elected leader, sending a full tablet report...
I20260927 21:37:57.987766 24134 ts_tablet_manager.cc:1428] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260927 21:37:57.990381 24135 raft_consensus.cc:348] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:57.990726 24135 raft_consensus.cc:374] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:37:57.990836 24135 raft_consensus.cc:729] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f91016e8d741eaa4468907fd62cceb, State: Initialized, Role: FOLLOWER
I20260927 21:37:57.991423 24135 consensus_queue.cc:260] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:57.992341 23853 heartbeater.cc:503] Master 127.3.186.254:43195 was elected leader, sending a full tablet report...
I20260927 21:37:57.992867 24135 ts_tablet_manager.cc:1428] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20260927 21:37:58.031666 23854 tablet.cc:2369] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:37:58.127306 24115 tablet.cc:2369] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:37:58.206255 23985 tablet.cc:2369] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:37:58.363260 24140 raft_consensus.cc:482] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:37:58.363539 24140 raft_consensus.cc:504] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:58.365015 24140 leader_election.cc:288] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1230b5e0eaba4b59b200aca1d582de04 (127.3.186.195:46013), f4116a32f1b74451b4812bce2b437047 (127.3.186.194:36513)
I20260927 21:37:58.373281 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.365560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:36301 (local address 127.3.186.195:46013)
0927 21:37:58.365758 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:58.365767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:58.366514 (+ 747us) server_negotiation.cc:407] Connection header received
0927 21:37:58.366651 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:58.366659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:58.366734 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:58.366814 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:58.367510 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.368540 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.371179 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.372249 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.372400 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.372472 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:58.372943 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:37:58.373057 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:37:58.373283 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.365560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.186.193:49455 (local address 127.3.186.194:36513)
0927 21:37:58.365940 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:58.365947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:58.366305 (+ 358us) server_negotiation.cc:407] Connection header received
0927 21:37:58.366519 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:58.366524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:58.366626 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:58.366720 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:37:58.367317 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.368400 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.371172 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.372282 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.372425 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.372479 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:58.372968 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 21:37:58.373073 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260927 21:37:58.373427 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.365356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.195:46013 (local address 127.3.186.193:36301)
0927 21:37:58.366416 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.366433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:58.366534 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:58.366843 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:58.366846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:58.366904 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:58.367346 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:58.367362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.368655 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.368665 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:58.370969 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:58.370983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.372498 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.372504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:58.372717 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.372786 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 21:37:58.372885 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:37:58.373006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":96,"spinlock_wait_cycles":9088,"thread_start_us":393,"threads_started":1}
I20260927 21:37:58.373772 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.365275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.194:36513 (local address 127.3.186.193:49455)
0927 21:37:58.366178 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.366207 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:37:58.366364 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:37:58.366742 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:37:58.366745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:37:58.366763 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:37:58.367177 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:58.367190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.368524 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.368527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:58.370969 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:37:58.370983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.372474 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.372486 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:37:58.372713 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.372787 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 21:37:58.372916 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:37:58.373009 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":160,"thread_start_us":831,"threads_started":1}
I20260927 21:37:58.374346 24069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0add7d6f8f47d186691e855d8821e4" candidate_uuid: "d4f91016e8d741eaa4468907fd62cceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1230b5e0eaba4b59b200aca1d582de04" is_pre_election: true
I20260927 21:37:58.374660 23939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0add7d6f8f47d186691e855d8821e4" candidate_uuid: "d4f91016e8d741eaa4468907fd62cceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4116a32f1b74451b4812bce2b437047" is_pre_election: true
I20260927 21:37:58.374861 24069 raft_consensus.cc:2454] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f91016e8d741eaa4468907fd62cceb in term 0.
I20260927 21:37:58.375193 23939 raft_consensus.cc:2454] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f91016e8d741eaa4468907fd62cceb in term 0.
I20260927 21:37:58.375521 23743 leader_election.cc:302] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1230b5e0eaba4b59b200aca1d582de04, d4f91016e8d741eaa4468907fd62cceb; no voters:
I20260927 21:37:58.375946 24140 raft_consensus.cc:2781] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:37:58.376087 24140 raft_consensus.cc:482] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:37:58.376201 24140 raft_consensus.cc:3037] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:58.378432 24140 raft_consensus.cc:504] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:58.379029 24140 leader_election.cc:288] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [CANDIDATE]: Term 1 election: Requested vote from peers 1230b5e0eaba4b59b200aca1d582de04 (127.3.186.195:46013), f4116a32f1b74451b4812bce2b437047 (127.3.186.194:36513)
I20260927 21:37:58.379395 24069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0add7d6f8f47d186691e855d8821e4" candidate_uuid: "d4f91016e8d741eaa4468907fd62cceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1230b5e0eaba4b59b200aca1d582de04"
I20260927 21:37:58.379438 23939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0add7d6f8f47d186691e855d8821e4" candidate_uuid: "d4f91016e8d741eaa4468907fd62cceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4116a32f1b74451b4812bce2b437047"
I20260927 21:37:58.379601 24069 raft_consensus.cc:3037] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:58.379601 23939 raft_consensus.cc:3037] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:37:58.382611 24069 raft_consensus.cc:2454] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f91016e8d741eaa4468907fd62cceb in term 1.
I20260927 21:37:58.382612 23939 raft_consensus.cc:2454] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f91016e8d741eaa4468907fd62cceb in term 1.
I20260927 21:37:58.382992 23744 leader_election.cc:302] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4f91016e8d741eaa4468907fd62cceb, f4116a32f1b74451b4812bce2b437047; no voters:
I20260927 21:37:58.383239 24140 raft_consensus.cc:2781] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:37:58.383644 24140 raft_consensus.cc:686] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 1 LEADER]: Becoming Leader. State: Replica: d4f91016e8d741eaa4468907fd62cceb, State: Running, Role: LEADER
I20260927 21:37:58.384081 24140 consensus_queue.cc:237] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } }
I20260927 21:37:58.388484 23666 catalog_manager.cc:5495] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb reported cstate change: term changed from 0 to 1, leader changed from <none> to d4f91016e8d741eaa4468907fd62cceb (127.3.186.193). New cstate: current_term: 1 leader_uuid: "d4f91016e8d741eaa4468907fd62cceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4f91016e8d741eaa4468907fd62cceb" member_type: VOTER last_known_addr { host: "127.3.186.193" port: 33991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 } health_report { overall_health: UNKNOWN } } }
I20260927 21:37:58.509147 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.502809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.3.186.193:33991)
0927 21:37:58.503277 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:58.503283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:58.503615 (+ 332us) server_negotiation.cc:407] Connection header received
0927 21:37:58.503797 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:58.503803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:58.503904 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:58.504069 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:37:58.504440 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.505472 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.506807 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.507417 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.507551 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.507958 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:37:58.508555 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:37:58.508701 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:58.508747 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 21:37:58.508821 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260927 21:37:58.516808 23939 raft_consensus.cc:1261] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 1 FOLLOWER]: Refusing update from remote peer d4f91016e8d741eaa4468907fd62cceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:58.516945 24069 raft_consensus.cc:1261] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 1 FOLLOWER]: Refusing update from remote peer d4f91016e8d741eaa4468907fd62cceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:37:58.517853 24143 consensus_queue.cc:1035] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1230b5e0eaba4b59b200aca1d582de04" member_type: VOTER last_known_addr { host: "127.3.186.195" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:58.518260 24140 consensus_queue.cc:1035] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4116a32f1b74451b4812bce2b437047" member_type: VOTER last_known_addr { host: "127.3.186.194" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:37:58.528529 24148 mvcc.cc:205] Tried to move back new op lower bound from 7334072641595572224 to 7334072641064382464. Current Snapshot: MvccSnapshot[applied={T|T < 7334072641595572224}]
I20260927 21:37:58.530303 24152 mvcc.cc:205] Tried to move back new op lower bound from 7334072641595572224 to 7334072641064382464. Current Snapshot: MvccSnapshot[applied={T|T < 7334072641595572224}]
I20260927 21:37:58.542037 23667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:58.542246 23667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:37:58.545213 23667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d180b74f7f5442118dff951aad7a6e00: Sending DeleteTablet for 3 replicas of tablet ea0add7d6f8f47d186691e855d8821e4
I20260927 21:37:58.546278 24049 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0add7d6f8f47d186691e855d8821e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:58 UTC) from {username='slave'} at 127.0.0.1:38170
I20260927 21:37:58.546669 23788 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0add7d6f8f47d186691e855d8821e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:58 UTC) from {username='slave'} at 127.0.0.1:45518
I20260927 21:37:58.547358 24159 tablet_replica.cc:330] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: stopping tablet replica
I20260927 21:37:58.547252 23919 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0add7d6f8f47d186691e855d8821e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:37:58 UTC) from {username='slave'} at 127.0.0.1:45762
I20260927 21:37:58.547772 24159 raft_consensus.cc:2229] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:58.548147 24159 raft_consensus.cc:2258] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:58.549062 24161 tablet_replica.cc:330] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: stopping tablet replica
I20260927 21:37:58.549403 24161 raft_consensus.cc:2229] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:37:58.549458 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23726
I20260927 21:37:58.549851 24161 raft_consensus.cc:2258] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:58.550099 24159 ts_tablet_manager.cc:1905] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:58.550566 24160 tablet_replica.cc:330] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: stopping tablet replica
I20260927 21:37:58.550896 24160 raft_consensus.cc:2229] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:37:58.551481 24160 raft_consensus.cc:2258] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:37:58.552980 24161 ts_tablet_manager.cc:1905] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:58.553148 24160 ts_tablet_manager.cc:1905] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:37:58.559908 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.551283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.3.186.193:33991)
0927 21:37:58.551488 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:58.551497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:58.551929 (+ 432us) server_negotiation.cc:407] Connection header received
0927 21:37:58.552134 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:58.552141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:58.552228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:58.552298 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:58.552953 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.554201 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.555962 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.556657 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.556758 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.558245 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:58.558259 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:58.558272 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:58.559033 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:58.559089 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:58.559114 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:58.559192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:58.559443 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:37:58.559535 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:37:58.560756 24160 ts_tablet_manager.cc:1918] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:58.560971 23848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:58.560969 24160 log.cc:1196] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-0/wal/wals/ea0add7d6f8f47d186691e855d8821e4
I20260927 21:37:58.562769 24159 ts_tablet_manager.cc:1918] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:58.562916 24159 log.cc:1196] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-2/wal/wals/ea0add7d6f8f47d186691e855d8821e4
I20260927 21:37:58.563681 24159 ts_tablet_manager.cc:1939] T ea0add7d6f8f47d186691e855d8821e4 P 1230b5e0eaba4b59b200aca1d582de04: Deleting consensus metadata
I20260927 21:37:58.565140 24161 ts_tablet_manager.cc:1918] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:37:58.565285 24161 log.cc:1196] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Deleting WAL directory at /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544773327888-3819-0/minicluster-data/ts-1/wal/wals/ea0add7d6f8f47d186691e855d8821e4
I20260927 21:37:58.565591 23653 catalog_manager.cc:4841] TS 1230b5e0eaba4b59b200aca1d582de04 (127.3.186.195:46013): tablet ea0add7d6f8f47d186691e855d8821e4 (table test-table [id=168401ca5d0243319e23d92992e23d5e]) successfully deleted
I20260927 21:37:58.565775 24161 ts_tablet_manager.cc:1939] T ea0add7d6f8f47d186691e855d8821e4 P f4116a32f1b74451b4812bce2b437047: Deleting consensus metadata
I20260927 21:37:58.567049 23654 catalog_manager.cc:4841] TS f4116a32f1b74451b4812bce2b437047 (127.3.186.194:36513): tablet ea0add7d6f8f47d186691e855d8821e4 (table test-table [id=168401ca5d0243319e23d92992e23d5e]) successfully deleted
I20260927 21:37:58.652022 24160 ts_tablet_manager.cc:1939] T ea0add7d6f8f47d186691e855d8821e4 P d4f91016e8d741eaa4468907fd62cceb: Deleting consensus metadata
I20260927 21:37:58.652064 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23726
W20260927 21:37:58.668824 23654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.186.193:33991 (error 108)
I20260927 21:37:58.669044 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23857
I20260927 21:37:58.671106 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.670230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45534)
0927 21:37:58.670700 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.670825 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":188,"threads_started":1}
W20260927 21:37:58.671375 23654 catalog_manager.cc:4584] TS d4f91016e8d741eaa4468907fd62cceb (127.3.186.193:33991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea0add7d6f8f47d186691e855d8821e4: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
I20260927 21:37:58.675596 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.669909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.3.186.194:36513)
0927 21:37:58.670088 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:58.670095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:58.670114 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:37:58.670273 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:58.670280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:58.670423 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:58.670536 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:58.670978 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.671822 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.672959 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.673331 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.673416 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.674367 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:58.674375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:58.674388 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:58.674922 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:58.674965 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:58.674984 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:58.675076 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:58.675185 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:37:58.675262 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:37:58.676182 23979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:58.724764 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.724303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45540)
0927 21:37:58.724513 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.724614 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:37:58.725793 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.725368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45544)
0927 21:37:58.725560 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.725634 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:37:58.762157 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23857
I20260927 21:37:58.779358 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23987
I20260927 21:37:58.785761 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.780031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.3.186.195:46013)
0927 21:37:58.780212 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:37:58.780219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:37:58.780239 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:37:58.780427 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:37:58.780431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:37:58.780522 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:37:58.780635 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:37:58.781023 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.781786 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.783029 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:37:58.783422 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:37:58.783552 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:37:58.784480 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:37:58.784490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:37:58.784504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:37:58.785036 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:37:58.785083 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:37:58.785101 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:37:58.785216 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:37:58.785340 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 21:37:58.785445 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:37:58.786166 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.785747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45552)
0927 21:37:58.785918 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.786017 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:37:58.786474 24109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:58.787359 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.786820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45566)
0927 21:37:58.787016 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.787145 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:37:58.867645 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.867179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45580)
0927 21:37:58.867408 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.867508 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 21:37:58.868793 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:37:58.868370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.193:33991 (local address 127.0.0.1:45592)
0927 21:37:58.868548 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:37:58.868643 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.193:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:37:58.872150 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23987
I20260927 21:37:58.888017 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 23636
I20260927 21:37:58.888940 23697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:37:58.990703 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 23636
2026-09-27T21:37:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2945 ms)
[----------] 6 tests from EncryptionPolicyTest (20277 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:37:59 dist-test-slave-pqr8 krb5kdc[24171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:37:59 dist-test-slave-pqr8 krb5kdc[24171](info): set up 2 sockets
Sep 27 21:37:59 dist-test-slave-pqr8 krb5kdc[24171](info): commencing operation
krb5kdc: starting...
W20260927 21:38:00.077838 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:00 dist-test-slave-pqr8 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:00.294696 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:00.601104 24184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:00.601356 24184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:00.601660 24184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:00.609277 24184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:00.609375 24184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:00.609477 24184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:00.609615 24184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:00.609701 24184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:00.619411 24184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:00.620491 24184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:00.621827 24184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:00.632776 24189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:00.632705 24192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:00.632750 24190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:00.633728 24184 server_base.cc:1034] running on GCE node
I20260927 21:38:00.634471 24184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:00.635720 24184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:00.636948 24184 hybrid_clock.cc:648] HybridClock initialized: now 1790545080636919 us; error 51 us; skew 500 ppm
I20260927 21:38:00.637276 24184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:00 dist-test-slave-pqr8 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:00.640352 24184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:00.641841 24184 webserver.cc:460] Webserver started at http://127.0.0.1:43523/ using document root <none> and password file <none>
I20260927 21:38:00.642498 24184 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:00.642679 24184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:00.642973 24184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:00.645745 24184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "eb7cdac2cefc49e99559fe480f53f0ce"
format_stamp: "Formatted at 2026-09-27 21:38:00 on dist-test-slave-pqr8"
server_key: "4db41e336c223a05dc87331ab8a9ff2e"
server_key_iv: "80258ae563d611eb599754435646a7f4"
server_key_version: "encryptionkey@0"
I20260927 21:38:00.646483 24184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "eb7cdac2cefc49e99559fe480f53f0ce"
format_stamp: "Formatted at 2026-09-27 21:38:00 on dist-test-slave-pqr8"
server_key: "4db41e336c223a05dc87331ab8a9ff2e"
server_key_iv: "80258ae563d611eb599754435646a7f4"
server_key_version: "encryptionkey@0"
I20260927 21:38:00.651115 24184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:38:00.654134 24199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:00.655186 24184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:00.655345 24184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "eb7cdac2cefc49e99559fe480f53f0ce"
format_stamp: "Formatted at 2026-09-27 21:38:00 on dist-test-slave-pqr8"
server_key: "4db41e336c223a05dc87331ab8a9ff2e"
server_key_iv: "80258ae563d611eb599754435646a7f4"
server_key_version: "encryptionkey@0"
I20260927 21:38:00.655520 24184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:00.693744 24184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:00.697798 24184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:00.698074 24184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:00.715862 24184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41279
I20260927 21:38:00.715909 24250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41279 every 8 connection(s)
I20260927 21:38:00.717077 24184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:00.720414 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:00.722668 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24184
I20260927 21:38:00.727999 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:00.728801 3819 external_mini_cluster.cc:1428] Setting key 679e34194608102ff6ad19309283d504
I20260927 21:38:00.733250 24251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce: Bootstrap starting.
I20260927 21:38:00.736898 24251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:00.738149 24251 log.cc:824] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:00.741264 24251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce: No bootstrap required, opened a new log
Sep 27 21:38:00 dist-test-slave-pqr8 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:38:00.748922 24251 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 21:38:00.749284 24251 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:00.749402 24251 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7cdac2cefc49e99559fe480f53f0ce, State: Initialized, Role: FOLLOWER
I20260927 21:38:00.750068 24251 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 21:38:00.750321 24251 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:00.750468 24251 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:00.750653 24251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:00.750993 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:00.730606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38011 (local address 127.0.0.1:41279)
0927 21:38:00.731424 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:00.731437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:00.731476 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:38:00.732983 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:00.733003 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:00.733328 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:00.733718 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:00.735429 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:00.736755 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:00.738589 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:00.739624 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:00.739757 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:00.744087 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:00.744094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:00.744105 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:00.744144 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:00.747042 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:00.747765 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:00.747775 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:00.747780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:00.747936 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:00.748403 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:00.748413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:00.748415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:00.748866 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:00.749189 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:00.749629 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 21:38:00.749909 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":431,"threads_started":1}
I20260927 21:38:00.753424 24251 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 21:38:00.754035 24251 leader_election.cc:302] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7cdac2cefc49e99559fe480f53f0ce; no voters:
I20260927 21:38:00.754666 24251 leader_election.cc:288] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:00.754727 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:00.754801 24256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:00.756958 24251 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:00.761675 24256 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [term 1 LEADER]: Becoming Leader. State: Replica: eb7cdac2cefc49e99559fe480f53f0ce, State: Running, Role: LEADER
I20260927 21:38:00.762346 24256 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 21:38:00.765338 24257 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } } }
I20260927 21:38:00.765779 24257 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:00.766526 24258 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7cdac2cefc49e99559fe480f53f0ce. Latest consensus state: current_term: 1 leader_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7cdac2cefc49e99559fe480f53f0ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } } }
I20260927 21:38:00.766860 24258 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:00.771102 24268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:00.776458 24268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:00.777184 24254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:00.760140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33877 (local address 127.0.0.1:41279)
0927 21:38:00.760430 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:00.760435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:00.760449 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:38:00.760494 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:00.760497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:00.760541 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:00.760638 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:00.762182 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:00.763015 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:00.764454 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:00.765120 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:00.765220 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:00.768939 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:00.768945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:00.768952 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:00.768999 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:00.773103 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:00.774115 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:00.774120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:00.774123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:00.774248 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:00.774792 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:00.774797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:00.774800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:00.775244 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:00.775510 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:00.776652 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0927 21:38:00.776814 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260927 21:38:00.784970 24268 catalog_manager.cc:1347] Generated new cluster ID: ae0fe3a274724cf2aca8c75aec0be6a2
I20260927 21:38:00.785059 24268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:00.799582 24268 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:00.800642 24268 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:00.805678 24268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7cdac2cefc49e99559fe480f53f0ce: Generated new TSK 0
I20260927 21:38:00.806331 24268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:00.819458 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24184
I20260927 21:38:00.820698 24246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:00.904079 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24184
W20260927 21:38:00.920786 24169 connection.cc:535] client connection to 127.0.0.1:41279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41279): Connection reset by peer (error 104)
2026-09-27T21:38:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:00 dist-test-slave-pqr8 krb5kdc[24284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:00 dist-test-slave-pqr8 krb5kdc[24284](info): set up 2 sockets
Sep 27 21:38:00 dist-test-slave-pqr8 krb5kdc[24284](info): commencing operation
krb5kdc: starting...
W20260927 21:38:01.969924 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:02 dist-test-slave-pqr8 krb5kdc[24284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:02.178938 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:02.480881 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:02.481124 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:02.481212 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:02.487989 24297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:02.488076 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:02.488127 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:02.488185 24297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:02.488227 24297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:02.496879 24297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:02.497871 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:02.498956 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:02.509822 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:02.509895 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:02.509963 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:02.510910 24297 server_base.cc:1034] running on GCE node
I20260927 21:38:02.511579 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:02.512751 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:02.513932 24297 hybrid_clock.cc:648] HybridClock initialized: now 1790545082513887 us; error 40 us; skew 500 ppm
I20260927 21:38:02.514235 24297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:02 dist-test-slave-pqr8 krb5kdc[24284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:02.517205 24297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:02.518527 24297 webserver.cc:460] Webserver started at http://127.0.0.1:40647/ using document root <none> and password file <none>
I20260927 21:38:02.519140 24297 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:02.519261 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:02.519544 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:02.522177 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "f0a74e69dd414204864e5d78ffb58fe5"
format_stamp: "Formatted at 2026-09-27 21:38:02 on dist-test-slave-pqr8"
server_key: "a11ab63fec7b91e2b3a78b0b976b6852"
server_key_iv: "da79238e8b39cecefbeddfbee51dc001"
server_key_version: "encryptionkey@0"
I20260927 21:38:02.522883 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "f0a74e69dd414204864e5d78ffb58fe5"
format_stamp: "Formatted at 2026-09-27 21:38:02 on dist-test-slave-pqr8"
server_key: "a11ab63fec7b91e2b3a78b0b976b6852"
server_key_iv: "da79238e8b39cecefbeddfbee51dc001"
server_key_version: "encryptionkey@0"
I20260927 21:38:02.527556 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:38:02.530615 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:02.531759 24297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:02.531927 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "f0a74e69dd414204864e5d78ffb58fe5"
format_stamp: "Formatted at 2026-09-27 21:38:02 on dist-test-slave-pqr8"
server_key: "a11ab63fec7b91e2b3a78b0b976b6852"
server_key_iv: "da79238e8b39cecefbeddfbee51dc001"
server_key_version: "encryptionkey@0"
I20260927 21:38:02.532101 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:02.559679 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:02.563657 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:02.563916 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:02.580744 24297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38051
I20260927 21:38:02.580781 24363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38051 every 8 connection(s)
I20260927 21:38:02.581939 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:02.585335 24364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:02.587457 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24297
I20260927 21:38:02.587636 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:02.588083 3819 external_mini_cluster.cc:1428] Setting key 8b309c15c651bbc8998da121bd414278
I20260927 21:38:02.598320 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5: Bootstrap starting.
I20260927 21:38:02.601903 24364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:02.602689 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:02.589878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42531 (local address 127.0.0.1:38051)
0927 21:38:02.590720 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:02.590734 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:02.590780 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:38:02.593101 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:02.593127 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:02.593619 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:02.594034 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:02.594529 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:02.596546 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:02.597844 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:02.598844 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:02.599012 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:02.600100 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:02.600109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:02.600128 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:02.600778 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:02.600871 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:02.600901 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:02.601057 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:02.601284 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 21:38:02.601429 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":361,"threads_started":1}
I20260927 21:38:02.603130 24364 log.cc:824] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:02.606118 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5: No bootstrap required, opened a new log
I20260927 21:38:02.606539 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:02.614346 24364 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a74e69dd414204864e5d78ffb58fe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38051 } }
I20260927 21:38:02.614746 24364 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:02.614843 24364 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a74e69dd414204864e5d78ffb58fe5, State: Initialized, Role: FOLLOWER
I20260927 21:38:02.615465 24364 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a74e69dd414204864e5d78ffb58fe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38051 } }
I20260927 21:38:02.615751 24364 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:02.615904 24364 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:02.616091 24364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:02.617136 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:02.610591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50407 (local address 127.0.0.1:38051)
0927 21:38:02.610788 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:02.610797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:02.611169 (+ 372us) server_negotiation.cc:407] Connection header received
0927 21:38:02.611336 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:02.611339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:02.611394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:02.611466 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:02.611983 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:02.612702 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:02.614476 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:02.615045 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:02.615163 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:02.616356 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:02.616360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:02.616363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:02.616381 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:02.616420 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:02.616435 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:02.616518 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:02.616741 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:38:02.616776 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:38:02.618805 24364 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a74e69dd414204864e5d78ffb58fe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38051 } }
I20260927 21:38:02.619243 24364 leader_election.cc:302] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a74e69dd414204864e5d78ffb58fe5; no voters:
I20260927 21:38:02.619750 24364 leader_election.cc:288] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:02.619838 24374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:02.620904 24374 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [term 1 LEADER]: Becoming Leader. State: Replica: f0a74e69dd414204864e5d78ffb58fe5, State: Running, Role: LEADER
I20260927 21:38:02.621371 24374 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a74e69dd414204864e5d78ffb58fe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38051 } }
I20260927 21:38:02.621670 24364 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:02.625427 24375 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a74e69dd414204864e5d78ffb58fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a74e69dd414204864e5d78ffb58fe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38051 } } }
I20260927 21:38:02.625844 24375 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:02.625577 24376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a74e69dd414204864e5d78ffb58fe5. Latest consensus state: current_term: 1 leader_uuid: "f0a74e69dd414204864e5d78ffb58fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a74e69dd414204864e5d78ffb58fe5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38051 } } }
I20260927 21:38:02.626025 24376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:02.628999 24383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:02.634543 24383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:02.642349 24383 catalog_manager.cc:1347] Generated new cluster ID: 266472af832646148a9ec8012f2fc780
I20260927 21:38:02.642447 24383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:02.660176 24383 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:02.661339 24383 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:02.667534 24383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a74e69dd414204864e5d78ffb58fe5: Generated new TSK 0
I20260927 21:38:02.668263 24383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:02.679442 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24297
W20260927 21:38:02.680891 24359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42531
W20260927 21:38:02.681844 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24297: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:02.682036 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24297
W20260927 21:38:02.698174 24282 connection.cc:535] client connection to 127.0.0.1:38051 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38051): Connection reset by peer (error 104)
2026-09-27T21:38:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:02 dist-test-slave-pqr8 krb5kdc[24395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:02 dist-test-slave-pqr8 krb5kdc[24395](info): set up 2 sockets
Sep 27 21:38:02 dist-test-slave-pqr8 krb5kdc[24395](info): commencing operation
krb5kdc: starting...
W20260927 21:38:03.737825 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:03 dist-test-slave-pqr8 krb5kdc[24395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:03.952303 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:04.255745 24408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:04.255976 24408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:04.256065 24408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:04.262784 24408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:04.262871 24408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:04.262913 24408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:04.262970 24408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:04.263012 24408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:04.271989 24408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:04.272965 24408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:04.274116 24408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:04.285354 24413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:04.285313 24414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:04.285646 24416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:04.286296 24408 server_base.cc:1034] running on GCE node
I20260927 21:38:04.286991 24408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:04.288188 24408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:04.289419 24408 hybrid_clock.cc:648] HybridClock initialized: now 1790545084289376 us; error 54 us; skew 500 ppm
I20260927 21:38:04.289762 24408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:04 dist-test-slave-pqr8 krb5kdc[24395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:04.292886 24408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:04.294183 24408 webserver.cc:460] Webserver started at http://127.0.0.1:42177/ using document root <none> and password file <none>
I20260927 21:38:04.294739 24408 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:04.294859 24408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:04.295099 24408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:04.297745 24408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "40fe631aef3e46269c05fcda6ac68d63"
format_stamp: "Formatted at 2026-09-27 21:38:04 on dist-test-slave-pqr8"
server_key: "27d63894a288268fa43cd2f546a2c57b"
server_key_iv: "cac8c2a65f18c3cbfe5235bbaebf3969"
server_key_version: "encryptionkey@0"
I20260927 21:38:04.298422 24408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "40fe631aef3e46269c05fcda6ac68d63"
format_stamp: "Formatted at 2026-09-27 21:38:04 on dist-test-slave-pqr8"
server_key: "27d63894a288268fa43cd2f546a2c57b"
server_key_iv: "cac8c2a65f18c3cbfe5235bbaebf3969"
server_key_version: "encryptionkey@0"
I20260927 21:38:04.303030 24408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:38:04.306141 24423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:04.307274 24408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:04.307399 24408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "40fe631aef3e46269c05fcda6ac68d63"
format_stamp: "Formatted at 2026-09-27 21:38:04 on dist-test-slave-pqr8"
server_key: "27d63894a288268fa43cd2f546a2c57b"
server_key_iv: "cac8c2a65f18c3cbfe5235bbaebf3969"
server_key_version: "encryptionkey@0"
I20260927 21:38:04.307541 24408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:04.312680 24408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:04.316180 24408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:04.316395 24408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:04.333784 24408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37037
I20260927 21:38:04.333873 24474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37037 every 8 connection(s)
I20260927 21:38:04.335023 24408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:04.338260 24475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:04.339501 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24408
I20260927 21:38:04.339685 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:04.340092 3819 external_mini_cluster.cc:1428] Setting key 0dfc12be88a20ca58e16f8df6c88ef51
I20260927 21:38:04.347795 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:04.341649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49629 (local address 127.0.0.1:37037)
0927 21:38:04.342528 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:04.342542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:04.342595 (+ 53us) server_negotiation.cc:407] Connection header received
0927 21:38:04.344398 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:04.344424 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:04.344808 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:04.345202 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:04.346106 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:04.346115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:04.346130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:04.346145 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:04.346200 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:04.346222 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:04.346420 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:04.346543 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:38:04.346707 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":375,"threads_started":1}
I20260927 21:38:04.351468 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:04.351909 24475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63: Bootstrap starting.
I20260927 21:38:04.354622 24475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:04.355865 24475 log.cc:824] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:04.357949 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:04.355569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46659 (local address 127.0.0.1:37037)
0927 21:38:04.355735 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:04.355741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:04.356115 (+ 374us) server_negotiation.cc:407] Connection header received
0927 21:38:04.356310 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:04.356314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:04.356357 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:04.356454 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:04.357308 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:04.357314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:04.357318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:04.357334 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:04.357376 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:04.357390 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:04.357500 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:04.357626 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 21:38:04.357699 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:38:04.358635 24475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63: No bootstrap required, opened a new log
I20260927 21:38:04.366402 24475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fe631aef3e46269c05fcda6ac68d63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260927 21:38:04.366707 24475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:04.366775 24475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fe631aef3e46269c05fcda6ac68d63, State: Initialized, Role: FOLLOWER
I20260927 21:38:04.367292 24475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fe631aef3e46269c05fcda6ac68d63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260927 21:38:04.367514 24475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:04.367631 24475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:04.367765 24475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:04.370106 24475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fe631aef3e46269c05fcda6ac68d63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260927 21:38:04.370492 24475 leader_election.cc:302] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fe631aef3e46269c05fcda6ac68d63; no voters:
I20260927 21:38:04.370970 24475 leader_election.cc:288] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:04.371094 24485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:04.371726 24485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [term 1 LEADER]: Becoming Leader. State: Replica: 40fe631aef3e46269c05fcda6ac68d63, State: Running, Role: LEADER
I20260927 21:38:04.372270 24485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fe631aef3e46269c05fcda6ac68d63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260927 21:38:04.372839 24475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:04.375030 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40fe631aef3e46269c05fcda6ac68d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fe631aef3e46269c05fcda6ac68d63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } } }
I20260927 21:38:04.375137 24487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fe631aef3e46269c05fcda6ac68d63. Latest consensus state: current_term: 1 leader_uuid: "40fe631aef3e46269c05fcda6ac68d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fe631aef3e46269c05fcda6ac68d63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } } }
I20260927 21:38:04.375567 24487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:04.375564 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:04.377838 24491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:04.382517 24491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:04.390643 24491 catalog_manager.cc:1347] Generated new cluster ID: f9d33ad847404a08b4be86ad1a98aa3c
I20260927 21:38:04.390738 24491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:04.442204 24491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:04.442675 24491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:04.449625 24491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40fe631aef3e46269c05fcda6ac68d63: Generated new TSK 0
I20260927 21:38:04.450454 24491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:04.456472 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24408
W20260927 21:38:04.457500 24470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49629
W20260927 21:38:04.458217 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:04.458344 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24408
2026-09-27T21:38:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1773 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:04 dist-test-slave-pqr8 krb5kdc[24506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:04 dist-test-slave-pqr8 krb5kdc[24506](info): set up 2 sockets
Sep 27 21:38:04 dist-test-slave-pqr8 krb5kdc[24506](info): commencing operation
krb5kdc: starting...
W20260927 21:38:05.513728 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:05 dist-test-slave-pqr8 krb5kdc[24506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:05.720613 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:06.025345 24519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:06.025645 24519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:06.025822 24519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:06.032567 24519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:06.032701 24519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:06.032810 24519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:06.032920 24519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:06.033018 24519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:06.042007 24519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:06.043054 24519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:06.044193 24519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:06.055305 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:06.055352 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:06.055348 24524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:06.055958 24519 server_base.cc:1034] running on GCE node
I20260927 21:38:06.056630 24519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:06.057976 24519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:06.059216 24519 hybrid_clock.cc:648] HybridClock initialized: now 1790545086059187 us; error 45 us; skew 500 ppm
I20260927 21:38:06.059569 24519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:06 dist-test-slave-pqr8 krb5kdc[24506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:06.067871 24519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:06.069257 24519 webserver.cc:460] Webserver started at http://127.0.0.1:39749/ using document root <none> and password file <none>
I20260927 21:38:06.069947 24519 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:06.070079 24519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:06.070367 24519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:06.073066 24519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "3da6c35a0f654f54b7b7e86d3049a45e"
format_stamp: "Formatted at 2026-09-27 21:38:06 on dist-test-slave-pqr8"
server_key: "0ed67590fd52fb8a8695a8f436a0daaf"
server_key_iv: "f587ccb0c4e759f3a3237cee081dd6d4"
server_key_version: "encryptionkey@0"
I20260927 21:38:06.073848 24519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "3da6c35a0f654f54b7b7e86d3049a45e"
format_stamp: "Formatted at 2026-09-27 21:38:06 on dist-test-slave-pqr8"
server_key: "0ed67590fd52fb8a8695a8f436a0daaf"
server_key_iv: "f587ccb0c4e759f3a3237cee081dd6d4"
server_key_version: "encryptionkey@0"
I20260927 21:38:06.078505 24519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 21:38:06.081822 24534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:06.082980 24519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:38:06.083154 24519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "3da6c35a0f654f54b7b7e86d3049a45e"
format_stamp: "Formatted at 2026-09-27 21:38:06 on dist-test-slave-pqr8"
server_key: "0ed67590fd52fb8a8695a8f436a0daaf"
server_key_iv: "f587ccb0c4e759f3a3237cee081dd6d4"
server_key_version: "encryptionkey@0"
I20260927 21:38:06.083333 24519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:06.109752 24519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:06.113788 24519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:06.114043 24519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:06.131181 24519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35745
I20260927 21:38:06.131219 24585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35745 every 8 connection(s)
I20260927 21:38:06.132360 24519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:06.135628 24586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:06.137828 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24519
I20260927 21:38:06.138013 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:06.138522 3819 external_mini_cluster.cc:1428] Setting key 24fc5fbad778d1a0acbf82de1c8af085
I20260927 21:38:06.147616 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e: Bootstrap starting.
I20260927 21:38:06.150557 24586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:06.151495 24586 log.cc:824] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:38:06 dist-test-slave-pqr8 krb5kdc[24506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:38:06.154388 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e: No bootstrap required, opened a new log
I20260927 21:38:06.159701 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:06.140201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39745 (local address 127.0.0.1:35745)
0927 21:38:06.140941 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:06.140950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:06.140988 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:38:06.142575 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:06.142594 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:06.142926 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:06.143197 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:06.144860 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:06.146177 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:06.147926 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:06.148528 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:06.148739 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:06.153051 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:06.153058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:06.153070 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:06.153104 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:06.156023 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:06.156624 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:06.156629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:06.156634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:06.156787 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:06.157156 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:06.157161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:06.157164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:06.157709 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:06.157995 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:06.158456 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 21:38:06.158820 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":355,"threads_started":1}
I20260927 21:38:06.162426 24586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35745 } }
I20260927 21:38:06.162863 24586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:06.162982 24586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da6c35a0f654f54b7b7e86d3049a45e, State: Initialized, Role: FOLLOWER
I20260927 21:38:06.163296 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:06.163622 24586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35745 } }
I20260927 21:38:06.163849 24586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:06.164001 24586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:06.164162 24586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:06.166293 24586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35745 } }
I20260927 21:38:06.166746 24586 leader_election.cc:302] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da6c35a0f654f54b7b7e86d3049a45e; no voters:
I20260927 21:38:06.167358 24586 leader_election.cc:288] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:06.167443 24595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:06.169004 24595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [term 1 LEADER]: Becoming Leader. State: Replica: 3da6c35a0f654f54b7b7e86d3049a45e, State: Running, Role: LEADER
I20260927 21:38:06.169663 24586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:06.169595 24595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35745 } }
I20260927 21:38:06.175299 24598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da6c35a0f654f54b7b7e86d3049a45e. Latest consensus state: current_term: 1 leader_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35745 } } }
I20260927 21:38:06.175735 24598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:06.176203 24596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da6c35a0f654f54b7b7e86d3049a45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35745 } } }
I20260927 21:38:06.176554 24596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:06.179638 24604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:06.186600 24604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:06.188271 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:06.167540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45831 (local address 127.0.0.1:35745)
0927 21:38:06.167710 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:06.167718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:06.169731 (+ 2013us) server_negotiation.cc:407] Connection header received
0927 21:38:06.169975 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:06.169980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:06.170048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:06.170149 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:06.171652 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:06.172309 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:06.173423 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:06.174104 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:06.174231 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:06.177604 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:06.177613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:06.177625 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:06.177669 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:06.184497 (+ 6828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:06.185253 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:06.185258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:06.185261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:06.185387 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:06.185928 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:06.185933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:06.185938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:06.186387 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:06.186612 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:06.187723 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0927 21:38:06.187919 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:38:06.196808 24604 catalog_manager.cc:1347] Generated new cluster ID: 7845a6bc179f4a79bf22f85c6236ca22
I20260927 21:38:06.196907 24604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:06.206183 24604 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:06.207585 24604 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:06.216248 24604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da6c35a0f654f54b7b7e86d3049a45e: Generated new TSK 0
I20260927 21:38:06.217047 24604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:06.226317 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24519
I20260927 21:38:06.227613 24581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:06.312714 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24519
2026-09-27T21:38:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:06 dist-test-slave-pqr8 krb5kdc[24619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:06 dist-test-slave-pqr8 krb5kdc[24619](info): set up 2 sockets
Sep 27 21:38:06 dist-test-slave-pqr8 krb5kdc[24619](info): commencing operation
krb5kdc: starting...
W20260927 21:38:07.377661 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:07 dist-test-slave-pqr8 krb5kdc[24619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:07.580502 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:07.888052 24632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:07.888298 24632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:07.888382 24632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:07.895167 24632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:07.895269 24632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:07.895313 24632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:07.895373 24632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:07.895414 24632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:07.904232 24632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:07.905197 24632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:07.906371 24632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:07.917217 24637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:07.917271 24638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:07.917325 24640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:07.918051 24632 server_base.cc:1034] running on GCE node
I20260927 21:38:07.918704 24632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:07.919844 24632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:07.921037 24632 hybrid_clock.cc:648] HybridClock initialized: now 1790545087921006 us; error 27 us; skew 500 ppm
I20260927 21:38:07.921322 24632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:07 dist-test-slave-pqr8 krb5kdc[24619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:07.924490 24632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:07.925957 24632 webserver.cc:460] Webserver started at http://127.0.0.1:35261/ using document root <none> and password file <none>
I20260927 21:38:07.926543 24632 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:07.926640 24632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:07.926877 24632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:07.929708 24632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3"
format_stamp: "Formatted at 2026-09-27 21:38:07 on dist-test-slave-pqr8"
server_key: "2370b1d33d969724432c89ac2006a761"
server_key_iv: "346c05a3bd5b1927678325442246c632"
server_key_version: "encryptionkey@0"
I20260927 21:38:07.930466 24632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3"
format_stamp: "Formatted at 2026-09-27 21:38:07 on dist-test-slave-pqr8"
server_key: "2370b1d33d969724432c89ac2006a761"
server_key_iv: "346c05a3bd5b1927678325442246c632"
server_key_version: "encryptionkey@0"
I20260927 21:38:07.935161 24632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 21:38:07.938373 24647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:07.939548 24632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:38:07.939724 24632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3"
format_stamp: "Formatted at 2026-09-27 21:38:07 on dist-test-slave-pqr8"
server_key: "2370b1d33d969724432c89ac2006a761"
server_key_iv: "346c05a3bd5b1927678325442246c632"
server_key_version: "encryptionkey@0"
I20260927 21:38:07.939942 24632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:07.971818 24632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:07.975790 24632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:07.976048 24632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:07.993427 24632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35527
I20260927 21:38:07.993467 24698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35527 every 8 connection(s)
I20260927 21:38:07.994623 24632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:07.997946 24699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:07.998225 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24632
I20260927 21:38:07.998399 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:07.998942 3819 external_mini_cluster.cc:1428] Setting key 095a9bf917bcbd0e6906a3860a2c8d4b
I20260927 21:38:08.010664 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3: Bootstrap starting.
I20260927 21:38:08.012009 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:08.000598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42447 (local address 127.0.0.1:35527)
0927 21:38:08.001422 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:08.001431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:08.001467 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:38:08.003745 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:08.003763 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:08.004104 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:08.004413 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:08.005002 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:08.006295 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:08.008177 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:08.008824 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:08.008943 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:08.010111 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:08.010118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:08.010129 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:08.010579 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:08.010645 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:08.010669 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:08.010805 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:08.010972 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:38:08.011201 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":390,"threads_started":1}
I20260927 21:38:08.013485 24699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:08.014636 24699 log.cc:824] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:08.015167 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:08.016986 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3: No bootstrap required, opened a new log
I20260927 21:38:08.024992 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:08.019229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46667 (local address 127.0.0.1:35527)
0927 21:38:08.019409 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:08.019414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:08.019756 (+ 342us) server_negotiation.cc:407] Connection header received
0927 21:38:08.019876 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:08.019882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:08.019945 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:08.020048 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:08.020478 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:08.021482 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:08.022679 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:08.023325 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:08.023456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:08.024320 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:08.024330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:08.024335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:08.024353 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:08.024391 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:08.024414 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:08.024498 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:08.024611 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 21:38:08.024691 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:38:08.025591 24699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260927 21:38:08.025976 24699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:08.026087 24699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa04b9aa0b04a0aa96c8d53e9a94ec3, State: Initialized, Role: FOLLOWER
I20260927 21:38:08.026594 24699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260927 21:38:08.026830 24699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:08.026979 24699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:08.027137 24699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:08.029088 24699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260927 21:38:08.029570 24699 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa04b9aa0b04a0aa96c8d53e9a94ec3; no voters:
I20260927 21:38:08.030084 24699 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:08.030206 24709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:08.030884 24709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa04b9aa0b04a0aa96c8d53e9a94ec3, State: Running, Role: LEADER
I20260927 21:38:08.031400 24709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260927 21:38:08.031965 24699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:08.034426 24711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa04b9aa0b04a0aa96c8d53e9a94ec3. Latest consensus state: current_term: 1 leader_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260927 21:38:08.034320 24710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa04b9aa0b04a0aa96c8d53e9a94ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260927 21:38:08.034964 24711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:08.034965 24710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:08.036329 24718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:08.041011 24718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:08.048842 24718 catalog_manager.cc:1347] Generated new cluster ID: ff6864011f18418eb7d66cce1832a037
I20260927 21:38:08.048964 24718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:08.056797 24718 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:08.058480 24718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:08.073068 24718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa04b9aa0b04a0aa96c8d53e9a94ec3: Generated new TSK 0
I20260927 21:38:08.073886 24718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:08.082809 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24632
W20260927 21:38:08.083789 24694 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42447
W20260927 21:38:08.084419 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24632: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:08.084561 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24632
2026-09-27T21:38:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1771 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:08 dist-test-slave-pqr8 krb5kdc[24730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:08 dist-test-slave-pqr8 krb5kdc[24730](info): set up 2 sockets
Sep 27 21:38:08 dist-test-slave-pqr8 krb5kdc[24730](info): commencing operation
krb5kdc: starting...
W20260927 21:38:09.145785 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:09 dist-test-slave-pqr8 krb5kdc[24730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:09.360280 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:09.665522 24743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:09.665882 24743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:09.666038 24743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:09.672919 24743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:09.673053 24743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:09.673156 24743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:09.673259 24743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:09.673347 24743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:09.682578 24743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:09.683669 24743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:09.684850 24743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:09.695935 24748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:09.696004 24749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:09.695963 24751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:09.697086 24743 server_base.cc:1034] running on GCE node
I20260927 21:38:09.697888 24743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:09.699144 24743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:09.700363 24743 hybrid_clock.cc:648] HybridClock initialized: now 1790545089700324 us; error 62 us; skew 500 ppm
I20260927 21:38:09.700685 24743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:09 dist-test-slave-pqr8 krb5kdc[24730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:09.703722 24743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:09.705072 24743 webserver.cc:460] Webserver started at http://127.0.0.1:46341/ using document root <none> and password file <none>
I20260927 21:38:09.705734 24743 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:09.705848 24743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:09.706091 24743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:09.708843 24743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "ee3d623778084fbea7897d710861c680"
format_stamp: "Formatted at 2026-09-27 21:38:09 on dist-test-slave-pqr8"
server_key: "4d092f5be1054544584a5f74d1a220d3"
server_key_iv: "47b99a0dc57ebf8e857f89fde47aad66"
server_key_version: "encryptionkey@0"
I20260927 21:38:09.709576 24743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "ee3d623778084fbea7897d710861c680"
format_stamp: "Formatted at 2026-09-27 21:38:09 on dist-test-slave-pqr8"
server_key: "4d092f5be1054544584a5f74d1a220d3"
server_key_iv: "47b99a0dc57ebf8e857f89fde47aad66"
server_key_version: "encryptionkey@0"
I20260927 21:38:09.714085 24743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 21:38:09.717067 24758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:09.718387 24743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:38:09.718547 24743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "ee3d623778084fbea7897d710861c680"
format_stamp: "Formatted at 2026-09-27 21:38:09 on dist-test-slave-pqr8"
server_key: "4d092f5be1054544584a5f74d1a220d3"
server_key_iv: "47b99a0dc57ebf8e857f89fde47aad66"
server_key_version: "encryptionkey@0"
I20260927 21:38:09.718731 24743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:09.724066 24743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:09.728189 24743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:09.728466 24743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:09.745810 24743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41547
I20260927 21:38:09.745891 24809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41547 every 8 connection(s)
I20260927 21:38:09.747068 24743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:09.748632 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24743
I20260927 21:38:09.748805 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:09.749367 3819 external_mini_cluster.cc:1428] Setting key 67230571cb2f6f6e7260755efb880af9
I20260927 21:38:09.751192 24810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:09.756354 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:09.750975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58585 (local address 127.0.0.1:41547)
0927 21:38:09.751705 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:09.751715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:09.751752 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:38:09.753258 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:09.753290 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:09.753674 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:09.753985 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:09.754946 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:09.754951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:09.754970 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:09.754993 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:09.755062 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:09.755080 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:09.755234 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:09.755342 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 21:38:09.755528 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":351,"threads_started":1}
I20260927 21:38:09.759855 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:09.762696 24810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680: Bootstrap starting.
I20260927 21:38:09.766201 24810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:09.766269 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:09.764047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56599 (local address 127.0.0.1:41547)
0927 21:38:09.764210 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:09.764215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:09.764492 (+ 277us) server_negotiation.cc:407] Connection header received
0927 21:38:09.764622 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:09.764625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:09.764663 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:09.764732 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:09.765620 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:09.765624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:09.765626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:09.765640 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:09.765671 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:09.765683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:09.765778 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:09.765943 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:38:09.765981 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:38:09.767278 24810 log.cc:824] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:09.769683 24810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680: No bootstrap required, opened a new log
I20260927 21:38:09.776192 24810 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3d623778084fbea7897d710861c680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41547 } }
I20260927 21:38:09.776528 24810 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:09.776620 24810 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3d623778084fbea7897d710861c680, State: Initialized, Role: FOLLOWER
I20260927 21:38:09.777165 24810 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3d623778084fbea7897d710861c680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41547 } }
I20260927 21:38:09.777379 24810 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:09.777516 24810 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:09.777714 24810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:09.779641 24810 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3d623778084fbea7897d710861c680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41547 } }
I20260927 21:38:09.780071 24810 leader_election.cc:302] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3d623778084fbea7897d710861c680; no voters:
I20260927 21:38:09.780575 24810 leader_election.cc:288] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:09.780694 24820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:09.781347 24820 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [term 1 LEADER]: Becoming Leader. State: Replica: ee3d623778084fbea7897d710861c680, State: Running, Role: LEADER
I20260927 21:38:09.781934 24820 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3d623778084fbea7897d710861c680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41547 } }
I20260927 21:38:09.782642 24810 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:09.789530 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee3d623778084fbea7897d710861c680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3d623778084fbea7897d710861c680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41547 } } }
I20260927 21:38:09.789932 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:09.795113 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3d623778084fbea7897d710861c680. Latest consensus state: current_term: 1 leader_uuid: "ee3d623778084fbea7897d710861c680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3d623778084fbea7897d710861c680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41547 } } }
I20260927 21:38:09.795462 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:09.795738 24825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:09.800365 24825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:09.808949 24825 catalog_manager.cc:1347] Generated new cluster ID: 60698cebbd6d44dc923a6b981718ff3a
I20260927 21:38:09.809057 24825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:09.818177 24825 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:09.818535 24825 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:09.824110 24825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee3d623778084fbea7897d710861c680: Generated new TSK 0
I20260927 21:38:09.824940 24825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:09.840119 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24743
W20260927 21:38:09.841192 24805 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58585
W20260927 21:38:09.841866 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24743: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:09.842006 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24743
2026-09-27T21:38:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:09 dist-test-slave-pqr8 krb5kdc[24841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:09 dist-test-slave-pqr8 krb5kdc[24841](info): set up 2 sockets
Sep 27 21:38:09 dist-test-slave-pqr8 krb5kdc[24841](info): commencing operation
krb5kdc: starting...
W20260927 21:38:10.905813 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:11 dist-test-slave-pqr8 krb5kdc[24841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:11.112664 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:11.415644 24854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:11.415874 24854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:11.415958 24854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:11.422647 24854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:11.422731 24854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:11.422782 24854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:11.422838 24854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:11.422880 24854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:11.431771 24854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:11.432683 24854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:11.433781 24854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:11.444185 24859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:11.444224 24860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:11.444307 24862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:11.445307 24854 server_base.cc:1034] running on GCE node
I20260927 21:38:11.446063 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:11.447330 24854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:11.448515 24854 hybrid_clock.cc:648] HybridClock initialized: now 1790545091448467 us; error 43 us; skew 500 ppm
I20260927 21:38:11.448854 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:11 dist-test-slave-pqr8 krb5kdc[24841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:11.457624 24854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:11.458982 24854 webserver.cc:460] Webserver started at http://127.0.0.1:39947/ using document root <none> and password file <none>
I20260927 21:38:11.459652 24854 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:11.459787 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:11.460072 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:11.462831 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "7a79d76a769a408d8911a5c10ef54c4b"
format_stamp: "Formatted at 2026-09-27 21:38:11 on dist-test-slave-pqr8"
server_key: "2ec196e28b3f5db23db236098c832143"
server_key_iv: "2b31871ed13e620ac8c70b437681c810"
server_key_version: "encryptionkey@0"
I20260927 21:38:11.463539 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "7a79d76a769a408d8911a5c10ef54c4b"
format_stamp: "Formatted at 2026-09-27 21:38:11 on dist-test-slave-pqr8"
server_key: "2ec196e28b3f5db23db236098c832143"
server_key_iv: "2b31871ed13e620ac8c70b437681c810"
server_key_version: "encryptionkey@0"
I20260927 21:38:11.468281 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:38:11.471439 24869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:11.472587 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:38:11.472750 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "7a79d76a769a408d8911a5c10ef54c4b"
format_stamp: "Formatted at 2026-09-27 21:38:11 on dist-test-slave-pqr8"
server_key: "2ec196e28b3f5db23db236098c832143"
server_key_iv: "2b31871ed13e620ac8c70b437681c810"
server_key_version: "encryptionkey@0"
I20260927 21:38:11.472929 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:11.485275 24854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:11.489315 24854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:11.489650 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:11.506912 24854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41753
I20260927 21:38:11.506950 24920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41753 every 8 connection(s)
I20260927 21:38:11.508091 24854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:11.509732 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24854
I20260927 21:38:11.509920 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:11.510357 3819 external_mini_cluster.cc:1428] Setting key 04ebbcc8a115779817981c23a6a90b69
I20260927 21:38:11.511819 24921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:38:11 dist-test-slave-pqr8 krb5kdc[24841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:38:11.524291 24921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b: Bootstrap starting.
I20260927 21:38:11.528095 24921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:11.529286 24921 log.cc:824] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:11.532338 24921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b: No bootstrap required, opened a new log
I20260927 21:38:11.532999 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:11.511996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:58463 (local address 127.0.0.1:41753)
0927 21:38:11.512731 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:11.512745 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:11.512789 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:38:11.514721 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:11.514744 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:11.515192 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:11.515594 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:11.516838 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:11.518835 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:11.520012 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:11.520992 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:11.521197 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:11.524944 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:11.524953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:11.524973 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:11.525011 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:11.528812 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:11.529504 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:11.529534 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:11.529542 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:11.529741 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:11.530309 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:11.530312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:11.530314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:11.530879 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:11.531165 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:11.531536 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:38:11.531821 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":292,"threads_started":1}
I20260927 21:38:11.536495 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:11.540527 24921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79d76a769a408d8911a5c10ef54c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41753 } }
I20260927 21:38:11.540915 24921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:11.541019 24921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79d76a769a408d8911a5c10ef54c4b, State: Initialized, Role: FOLLOWER
I20260927 21:38:11.541651 24921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79d76a769a408d8911a5c10ef54c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41753 } }
I20260927 21:38:11.541942 24921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:11.542083 24921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:11.542250 24921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:11.544936 24921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79d76a769a408d8911a5c10ef54c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41753 } }
I20260927 21:38:11.545570 24921 leader_election.cc:302] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a79d76a769a408d8911a5c10ef54c4b; no voters:
I20260927 21:38:11.546288 24921 leader_election.cc:288] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:11.546406 24931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:11.547386 24931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [term 1 LEADER]: Becoming Leader. State: Replica: 7a79d76a769a408d8911a5c10ef54c4b, State: Running, Role: LEADER
I20260927 21:38:11.547974 24931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79d76a769a408d8911a5c10ef54c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41753 } }
I20260927 21:38:11.548655 24921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:11.549875 24932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a79d76a769a408d8911a5c10ef54c4b. Latest consensus state: current_term: 1 leader_uuid: "7a79d76a769a408d8911a5c10ef54c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79d76a769a408d8911a5c10ef54c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41753 } } }
I20260927 21:38:11.550371 24932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:11.556818 24933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a79d76a769a408d8911a5c10ef54c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79d76a769a408d8911a5c10ef54c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41753 } } }
I20260927 21:38:11.558013 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:11.540456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:36017 (local address 127.0.0.1:41753)
0927 21:38:11.540634 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:11.540643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:11.541181 (+ 538us) server_negotiation.cc:407] Connection header received
0927 21:38:11.541365 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:11.541373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:11.541447 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:11.541564 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:11.543256 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:11.544017 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:11.545855 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:11.546303 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:11.546469 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:11.549116 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:11.549123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:11.549133 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:11.549168 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:11.553003 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:11.554197 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:11.554205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:11.554211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:11.554397 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:11.555078 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:11.555085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:11.555091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:11.555530 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:11.557039 (+ 1509us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:11.557253 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:38:11.557485 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:38:11.558471 24933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:11.564018 24936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:11.569990 24936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:11.579886 24936 catalog_manager.cc:1347] Generated new cluster ID: b3c6c60fb46e44fd97472e88baff9084
I20260927 21:38:11.579993 24936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:11.591101 24936 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:11.592245 24936 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:11.599632 24936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a79d76a769a408d8911a5c10ef54c4b: Generated new TSK 0
I20260927 21:38:11.600274 24936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:11.608419 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24854
I20260927 21:38:11.610220 24916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:11.694928 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24854
W20260927 21:38:11.711534 24839 connection.cc:535] client connection to 127.0.0.1:41753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41753): Connection reset by peer (error 104)
2026-09-27T21:38:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:11 dist-test-slave-pqr8 krb5kdc[24953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:11 dist-test-slave-pqr8 krb5kdc[24953](info): set up 2 sockets
Sep 27 21:38:11 dist-test-slave-pqr8 krb5kdc[24953](info): commencing operation
krb5kdc: starting...
W20260927 21:38:12.757694 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:12 dist-test-slave-pqr8 krb5kdc[24953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:12.966887 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:13.270123 24966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:13.270448 24966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:13.270607 24966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:13.277295 24966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:13.277431 24966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:13.277535 24966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:13.277634 24966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:13.277707 24966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:13.286715 24966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:13.287822 24966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:13.289033 24966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:13.299896 24971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:13.299908 24974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:13.299866 24972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:13.300585 24966 server_base.cc:1034] running on GCE node
I20260927 21:38:13.301247 24966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:13.302479 24966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:13.303658 24966 hybrid_clock.cc:648] HybridClock initialized: now 1790545093303613 us; error 46 us; skew 500 ppm
I20260927 21:38:13.303961 24966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:13 dist-test-slave-pqr8 krb5kdc[24953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:13.306912 24966 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:13.308199 24966 webserver.cc:460] Webserver started at http://127.0.0.1:34923/ using document root <none> and password file <none>
I20260927 21:38:13.308813 24966 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:13.308943 24966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:13.309226 24966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:13.311858 24966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "96e75cdde4fb496f97a9388dd3324f51"
format_stamp: "Formatted at 2026-09-27 21:38:13 on dist-test-slave-pqr8"
server_key: "37782906a8ce04d668b3a186a883f66d"
server_key_iv: "7cc76c1449a136611215b96ae58c47f0"
server_key_version: "encryptionkey@0"
I20260927 21:38:13.312546 24966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "96e75cdde4fb496f97a9388dd3324f51"
format_stamp: "Formatted at 2026-09-27 21:38:13 on dist-test-slave-pqr8"
server_key: "37782906a8ce04d668b3a186a883f66d"
server_key_iv: "7cc76c1449a136611215b96ae58c47f0"
server_key_version: "encryptionkey@0"
I20260927 21:38:13.317076 24966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 21:38:13.320241 24981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:13.321378 24966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:38:13.321604 24966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "96e75cdde4fb496f97a9388dd3324f51"
format_stamp: "Formatted at 2026-09-27 21:38:13 on dist-test-slave-pqr8"
server_key: "37782906a8ce04d668b3a186a883f66d"
server_key_iv: "7cc76c1449a136611215b96ae58c47f0"
server_key_version: "encryptionkey@0"
I20260927 21:38:13.321787 24966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:13.346648 24966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:13.350519 24966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:13.350769 24966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:13.367386 24966 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33961
I20260927 21:38:13.367424 25032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33961 every 8 connection(s)
I20260927 21:38:13.368556 24966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:13.371770 25033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:13.373260 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 24966
I20260927 21:38:13.373446 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:13.373829 3819 external_mini_cluster.cc:1428] Setting key 1d52032c82e42efc42998bac82a9dc47
I20260927 21:38:13.383843 25033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51: Bootstrap starting.
I20260927 21:38:13.385926 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:13.375379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:53275 (local address 127.0.0.1:33961)
0927 21:38:13.376188 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:13.376197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:13.376232 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:38:13.377695 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:13.377713 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:13.378103 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:13.378384 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:13.378905 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:13.380275 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:13.382049 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:13.382644 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:13.382759 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:13.383868 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:13.383876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:13.383887 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:13.384389 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:13.384459 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:13.384481 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:13.384593 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:13.384851 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 21:38:13.385015 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":384,"threads_started":1}
I20260927 21:38:13.386770 25033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:13.388047 25033 log.cc:824] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:13.389621 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:13.390636 25033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51: No bootstrap required, opened a new log
I20260927 21:38:13.397826 25033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e75cdde4fb496f97a9388dd3324f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260927 21:38:13.398159 25033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:13.398254 25033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e75cdde4fb496f97a9388dd3324f51, State: Initialized, Role: FOLLOWER
I20260927 21:38:13.398828 25033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e75cdde4fb496f97a9388dd3324f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260927 21:38:13.399133 25033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:13.399192 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:13.393588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:52729 (local address 127.0.0.1:33961)
0927 21:38:13.393751 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:13.393758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:13.394108 (+ 350us) server_negotiation.cc:407] Connection header received
0927 21:38:13.394270 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:13.394274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:13.394330 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:13.394422 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:13.394870 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:13.395566 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:13.396788 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:13.397221 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:13.397344 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:13.398434 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:13.398439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:13.398442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:13.398457 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:13.398506 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:13.398529 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:13.398659 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:13.398846 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 21:38:13.398881 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:38:13.399304 25033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:13.399494 25033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:13.401914 25033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e75cdde4fb496f97a9388dd3324f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260927 21:38:13.402354 25033 leader_election.cc:302] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e75cdde4fb496f97a9388dd3324f51; no voters:
I20260927 21:38:13.402870 25033 leader_election.cc:288] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:13.402967 25043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:13.403707 25043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [term 1 LEADER]: Becoming Leader. State: Replica: 96e75cdde4fb496f97a9388dd3324f51, State: Running, Role: LEADER
I20260927 21:38:13.404187 25043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e75cdde4fb496f97a9388dd3324f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } }
I20260927 21:38:13.404672 25033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:13.406711 25045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e75cdde4fb496f97a9388dd3324f51. Latest consensus state: current_term: 1 leader_uuid: "96e75cdde4fb496f97a9388dd3324f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e75cdde4fb496f97a9388dd3324f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } } }
I20260927 21:38:13.407094 25045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:13.408643 25044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96e75cdde4fb496f97a9388dd3324f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e75cdde4fb496f97a9388dd3324f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33961 } } }
I20260927 21:38:13.408878 25049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:13.408902 25044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:13.415169 25049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:13.423044 25049 catalog_manager.cc:1347] Generated new cluster ID: 96b2d7502f024885a487e1adcf88b45d
I20260927 21:38:13.423141 25049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:13.442353 25049 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:13.443879 25049 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:13.462540 25049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96e75cdde4fb496f97a9388dd3324f51: Generated new TSK 0
I20260927 21:38:13.463315 25049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:13.469084 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24966
W20260927 21:38:13.470320 25028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.107:53275
W20260927 21:38:13.471022 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 24966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:13.471158 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 24966
W20260927 21:38:13.486784 24951 connection.cc:535] client connection to 127.0.0.1:33961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33961): Connection reset by peer (error 104)
2026-09-27T21:38:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1774 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:13 dist-test-slave-pqr8 krb5kdc[25064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:13 dist-test-slave-pqr8 krb5kdc[25064](info): set up 2 sockets
Sep 27 21:38:13 dist-test-slave-pqr8 krb5kdc[25064](info): commencing operation
krb5kdc: starting...
W20260927 21:38:14.533844 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:14 dist-test-slave-pqr8 krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:14.742899 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:15.045361 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:15.045606 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:15.045689 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:15.052511 25077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:15.052598 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:15.052641 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:15.052699 25077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:15.052767 25077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:15.061664 25077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:15.062641 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:15.063794 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:15.074513 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:15.075110 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:15.075136 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:15.075444 25077 server_base.cc:1034] running on GCE node
I20260927 21:38:15.076128 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:15.077416 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:15.078634 25077 hybrid_clock.cc:648] HybridClock initialized: now 1790545095078590 us; error 41 us; skew 500 ppm
I20260927 21:38:15.078975 25077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:15 dist-test-slave-pqr8 krb5kdc[25064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:15.081827 25077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:15.083104 25077 webserver.cc:460] Webserver started at http://127.0.0.1:39515/ using document root <none> and password file <none>
I20260927 21:38:15.083704 25077 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:15.083859 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:15.084142 25077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:15.087446 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "8bdabf6573a34dff804ab93f6e9bbd58"
format_stamp: "Formatted at 2026-09-27 21:38:15 on dist-test-slave-pqr8"
server_key: "0651583328d71a55f55c783bb8af1866"
server_key_iv: "8c549f3c9f9a694806bfd07eed2a7cb3"
server_key_version: "encryptionkey@0"
I20260927 21:38:15.088158 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "8bdabf6573a34dff804ab93f6e9bbd58"
format_stamp: "Formatted at 2026-09-27 21:38:15 on dist-test-slave-pqr8"
server_key: "0651583328d71a55f55c783bb8af1866"
server_key_iv: "8c549f3c9f9a694806bfd07eed2a7cb3"
server_key_version: "encryptionkey@0"
I20260927 21:38:15.092754 25077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 21:38:15.095757 25092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:15.096895 25077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:38:15.097067 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "8bdabf6573a34dff804ab93f6e9bbd58"
format_stamp: "Formatted at 2026-09-27 21:38:15 on dist-test-slave-pqr8"
server_key: "0651583328d71a55f55c783bb8af1866"
server_key_iv: "8c549f3c9f9a694806bfd07eed2a7cb3"
server_key_version: "encryptionkey@0"
I20260927 21:38:15.097286 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:15.102373 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:15.105943 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:15.106220 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:15.122840 25077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33329
I20260927 21:38:15.122905 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33329 every 8 connection(s)
I20260927 21:38:15.123980 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:15.127215 25144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:15.129798 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25077
I20260927 21:38:15.129972 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:15.130492 3819 external_mini_cluster.cc:1428] Setting key 2c7b721902fd307fdf7652119285324c
I20260927 21:38:15.137751 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:15.132174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:35127 (local address 127.0.0.1:33329)
0927 21:38:15.132928 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:15.132938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:15.132985 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:38:15.134630 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:15.134655 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:15.134961 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:15.135270 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:15.136268 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:15.136274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:15.136285 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:15.136303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:15.136366 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:15.136386 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:15.136548 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:15.136659 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:38:15.136903 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":354,"threads_started":1}
I20260927 21:38:15.138623 25144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58: Bootstrap starting.
I20260927 21:38:15.140965 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:15.141613 25144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:15.142746 25144 log.cc:824] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:15.145689 25144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58: No bootstrap required, opened a new log
I20260927 21:38:15.148048 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:15.145834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:55561 (local address 127.0.0.1:33329)
0927 21:38:15.145988 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:15.145995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:15.146392 (+ 397us) server_negotiation.cc:407] Connection header received
0927 21:38:15.146489 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:15.146494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:15.146541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:15.146633 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:15.147450 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:15.147455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:15.147460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:15.147476 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:15.147522 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:15.147540 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:15.147653 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:15.147705 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:38:15.147757 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:38:15.153659 25144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33329 } }
I20260927 21:38:15.153946 25144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:15.154012 25144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bdabf6573a34dff804ab93f6e9bbd58, State: Initialized, Role: FOLLOWER
I20260927 21:38:15.154479 25144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33329 } }
I20260927 21:38:15.154738 25144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:15.154960 25144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:15.155148 25144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:15.157155 25144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33329 } }
I20260927 21:38:15.157634 25144 leader_election.cc:302] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bdabf6573a34dff804ab93f6e9bbd58; no voters:
I20260927 21:38:15.158130 25144 leader_election.cc:288] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:15.158258 25154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:15.158990 25154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [term 1 LEADER]: Becoming Leader. State: Replica: 8bdabf6573a34dff804ab93f6e9bbd58, State: Running, Role: LEADER
I20260927 21:38:15.159497 25154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33329 } }
I20260927 21:38:15.159969 25144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:15.162106 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bdabf6573a34dff804ab93f6e9bbd58. Latest consensus state: current_term: 1 leader_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33329 } } }
I20260927 21:38:15.162539 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:15.163319 25155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bdabf6573a34dff804ab93f6e9bbd58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33329 } } }
I20260927 21:38:15.163745 25155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:15.164086 25159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:15.168785 25159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:15.176051 25159 catalog_manager.cc:1347] Generated new cluster ID: 0fefe746db6249f193fef53018278c1f
I20260927 21:38:15.176177 25159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:15.192504 25159 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:15.192900 25159 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:15.202278 25159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bdabf6573a34dff804ab93f6e9bbd58: Generated new TSK 0
I20260927 21:38:15.203079 25159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:15.211722 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25077
W20260927 21:38:15.212692 25139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.107:35127
W20260927 21:38:15.213253 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25077: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:15.213388 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25077
2026-09-27T21:38:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1742 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:15 dist-test-slave-pqr8 krb5kdc[25175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:15 dist-test-slave-pqr8 krb5kdc[25175](info): set up 2 sockets
Sep 27 21:38:15 dist-test-slave-pqr8 krb5kdc[25175](info): commencing operation
krb5kdc: starting...
W20260927 21:38:16.281819 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:16 dist-test-slave-pqr8 krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:16.486409 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:16.788152 25188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:16.788430 25188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:16.788586 25188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:16.795256 25188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:16.795390 25188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:16.795480 25188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:16.795573 25188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:16.795640 25188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:16.804627 25188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:16.805805 25188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:16.806960 25188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:16.817786 25194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:16.817725 25196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:16.818274 25193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:16.818706 25188 server_base.cc:1034] running on GCE node
I20260927 21:38:16.819416 25188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:16.820629 25188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:16.821833 25188 hybrid_clock.cc:648] HybridClock initialized: now 1790545096821785 us; error 37 us; skew 500 ppm
I20260927 21:38:16.822165 25188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:16 dist-test-slave-pqr8 krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:16.830256 25188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:16.831431 25188 webserver.cc:460] Webserver started at http://127.0.0.1:42985/ using document root <none> and password file <none>
I20260927 21:38:16.832005 25188 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:16.832140 25188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:16.832409 25188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:16.835223 25188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "0064d25e178049e89fa939c245c4da5a"
format_stamp: "Formatted at 2026-09-27 21:38:16 on dist-test-slave-pqr8"
server_key: "75c727e0ecccc22309a1bfe5d41da8ee"
server_key_iv: "a3e0abc39f0748e1fb46118ce72bc39c"
server_key_version: "encryptionkey@0"
I20260927 21:38:16.835861 25188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "0064d25e178049e89fa939c245c4da5a"
format_stamp: "Formatted at 2026-09-27 21:38:16 on dist-test-slave-pqr8"
server_key: "75c727e0ecccc22309a1bfe5d41da8ee"
server_key_iv: "a3e0abc39f0748e1fb46118ce72bc39c"
server_key_version: "encryptionkey@0"
I20260927 21:38:16.840492 25188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 21:38:16.844048 25203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:16.845181 25188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:38:16.845317 25188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "0064d25e178049e89fa939c245c4da5a"
format_stamp: "Formatted at 2026-09-27 21:38:16 on dist-test-slave-pqr8"
server_key: "75c727e0ecccc22309a1bfe5d41da8ee"
server_key_iv: "a3e0abc39f0748e1fb46118ce72bc39c"
server_key_version: "encryptionkey@0"
I20260927 21:38:16.845465 25188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:16.858971 25188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:16.862383 25188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:16.862658 25188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:16.879448 25188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39889
I20260927 21:38:16.879503 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39889 every 8 connection(s)
I20260927 21:38:16.880550 25188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:16.883011 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25188
I20260927 21:38:16.883173 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:16.883581 3819 external_mini_cluster.cc:1428] Setting key 5fed0dcac6e6e809238b95cffe3782c4
I20260927 21:38:16.883868 25255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:16.896991 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a: Bootstrap starting.
Sep 27 21:38:16 dist-test-slave-pqr8 krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:38:16.899827 25255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:16.901114 25255 log.cc:824] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:16.904086 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a: No bootstrap required, opened a new log
I20260927 21:38:16.906257 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:16.885087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:58773 (local address 127.0.0.1:39889)
0927 21:38:16.885831 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:16.885840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:16.885876 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:38:16.887936 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:16.887954 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:16.888315 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:16.888597 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:16.890330 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:16.891466 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:16.893182 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:16.893873 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:16.893990 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:16.898369 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:16.898377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:16.898388 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:16.898430 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:16.901932 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:16.902751 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:16.902762 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:16.902770 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:16.902973 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:16.903399 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:16.903403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:16.903406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:16.903998 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:16.904282 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:16.904744 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:38:16.905041 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":320,"threads_started":1}
I20260927 21:38:16.909605 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:16.911919 25255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0064d25e178049e89fa939c245c4da5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260927 21:38:16.912309 25255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:16.912428 25255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0064d25e178049e89fa939c245c4da5a, State: Initialized, Role: FOLLOWER
I20260927 21:38:16.912966 25255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0064d25e178049e89fa939c245c4da5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260927 21:38:16.913210 25255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:16.913378 25255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:16.913656 25255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:16.915764 25255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0064d25e178049e89fa939c245c4da5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260927 21:38:16.916219 25255 leader_election.cc:302] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0064d25e178049e89fa939c245c4da5a; no voters:
I20260927 21:38:16.916846 25255 leader_election.cc:288] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:16.916946 25265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:16.919147 25255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:16.921794 25265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [term 1 LEADER]: Becoming Leader. State: Replica: 0064d25e178049e89fa939c245c4da5a, State: Running, Role: LEADER
I20260927 21:38:16.922544 25265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0064d25e178049e89fa939c245c4da5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } }
I20260927 21:38:16.926005 25266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0064d25e178049e89fa939c245c4da5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0064d25e178049e89fa939c245c4da5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } } }
I20260927 21:38:16.926417 25266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:16.927193 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:16.913599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:45609 (local address 127.0.0.1:39889)
0927 21:38:16.913751 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:16.913758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:16.914135 (+ 377us) server_negotiation.cc:407] Connection header received
0927 21:38:16.914306 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:16.914311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:16.914375 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:16.914474 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:16.915800 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:16.916614 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:16.918309 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:16.918724 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:16.918878 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:16.921162 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:16.921167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:16.921170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:16.921196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:16.923931 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:16.924727 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:16.924732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:16.924735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:16.924862 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:16.925454 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:16.925458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:16.925462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:16.925887 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:16.926141 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:16.926796 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 21:38:16.926949 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:38:16.929683 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0064d25e178049e89fa939c245c4da5a. Latest consensus state: current_term: 1 leader_uuid: "0064d25e178049e89fa939c245c4da5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0064d25e178049e89fa939c245c4da5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39889 } } }
I20260927 21:38:16.929910 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:16.934506 25272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:16.938445 25272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:16.945724 25272 catalog_manager.cc:1347] Generated new cluster ID: 2d47be7318c94d6fae9d44d6a2b34db6
I20260927 21:38:16.945828 25272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:16.973661 25272 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:16.975211 25272 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:16.983008 25272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0064d25e178049e89fa939c245c4da5a: Generated new TSK 0
I20260927 21:38:16.983789 25272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:16.991008 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25188
I20260927 21:38:16.991904 25250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:17.079206 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25188
W20260927 21:38:17.096087 25173 connection.cc:535] client connection to 127.0.0.1:39889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39889): Connection reset by peer (error 104)
2026-09-27T21:38:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:17 dist-test-slave-pqr8 krb5kdc[25287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:17 dist-test-slave-pqr8 krb5kdc[25287](info): set up 2 sockets
Sep 27 21:38:17 dist-test-slave-pqr8 krb5kdc[25287](info): commencing operation
krb5kdc: starting...
W20260927 21:38:18.137933 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:18 dist-test-slave-pqr8 krb5kdc[25287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:18.346782 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:18.648518 25300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:18.648761 25300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:18.648847 25300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:18.655550 25300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:18.655637 25300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:18.655679 25300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:18.655740 25300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:18.655781 25300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:18.664680 25300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:18.665647 25300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:18.666754 25300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:18.677495 25306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:18.677558 25308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:18.677601 25305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:18.682940 25300 server_base.cc:1034] running on GCE node
I20260927 21:38:18.683593 25300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:18.684767 25300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:18.685968 25300 hybrid_clock.cc:648] HybridClock initialized: now 1790545098685928 us; error 51 us; skew 500 ppm
I20260927 21:38:18.686270 25300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:18 dist-test-slave-pqr8 krb5kdc[25287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:18.689093 25300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:18.690394 25300 webserver.cc:460] Webserver started at http://127.0.0.1:46791/ using document root <none> and password file <none>
I20260927 21:38:18.690963 25300 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:18.691054 25300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:18.691268 25300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:18.694005 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "a4e7d7ca290c45348e00493c00de6262"
format_stamp: "Formatted at 2026-09-27 21:38:18 on dist-test-slave-pqr8"
server_key: "867696ed9c988e83ceefaf7990ef88c6"
server_key_iv: "2dbd931b9a4880dcdbb1d2d328239196"
server_key_version: "encryptionkey@0"
I20260927 21:38:18.694645 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "a4e7d7ca290c45348e00493c00de6262"
format_stamp: "Formatted at 2026-09-27 21:38:18 on dist-test-slave-pqr8"
server_key: "867696ed9c988e83ceefaf7990ef88c6"
server_key_iv: "2dbd931b9a4880dcdbb1d2d328239196"
server_key_version: "encryptionkey@0"
I20260927 21:38:18.699218 25300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:38:18.702359 25315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:18.703552 25300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:38:18.703676 25300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "a4e7d7ca290c45348e00493c00de6262"
format_stamp: "Formatted at 2026-09-27 21:38:18 on dist-test-slave-pqr8"
server_key: "867696ed9c988e83ceefaf7990ef88c6"
server_key_iv: "2dbd931b9a4880dcdbb1d2d328239196"
server_key_version: "encryptionkey@0"
I20260927 21:38:18.703830 25300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:18.716835 25300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:18.720443 25300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:18.720643 25300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:18.738884 25300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33819
I20260927 21:38:18.738955 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33819 every 8 connection(s)
I20260927 21:38:18.740134 25300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:18.743578 25367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:18.743834 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25300
I20260927 21:38:18.743991 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:18.744403 3819 external_mini_cluster.cc:1428] Setting key ac5cbcc7b6b2a4a9e4c58553bac5a2ec
I20260927 21:38:18.756219 25367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262: Bootstrap starting.
I20260927 21:38:18.756445 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:18.745909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:54007 (local address 127.0.0.1:33819)
0927 21:38:18.746677 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:18.746687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:18.746724 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:38:18.748179 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:18.748198 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:18.748560 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:18.748865 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:18.749425 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:18.750719 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:18.752493 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:18.753109 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:18.753232 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:18.754424 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:18.754431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:18.754445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:18.754919 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:18.754983 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:18.755004 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:18.755115 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:18.755385 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 21:38:18.755562 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":390,"threads_started":1}
I20260927 21:38:18.759611 25367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:18.759955 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:18.760799 25367 log.cc:824] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:18.763717 25367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262: No bootstrap required, opened a new log
I20260927 21:38:18.771129 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:18.764589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:37569 (local address 127.0.0.1:33819)
0927 21:38:18.764749 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:18.764755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:18.765295 (+ 540us) server_negotiation.cc:407] Connection header received
0927 21:38:18.765399 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:18.765403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:18.765474 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:18.765587 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:18.766077 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:18.767206 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:18.768495 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:18.769185 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:18.769344 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:18.770383 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:18.770389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:18.770393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:18.770411 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:18.770448 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:18.770468 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:18.770592 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:18.770775 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:38:18.770825 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:38:18.772511 25367 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e7d7ca290c45348e00493c00de6262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260927 21:38:18.772842 25367 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:18.772941 25367 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e7d7ca290c45348e00493c00de6262, State: Initialized, Role: FOLLOWER
I20260927 21:38:18.773427 25367 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e7d7ca290c45348e00493c00de6262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260927 21:38:18.773721 25367 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:18.773871 25367 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:18.774051 25367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:18.775975 25367 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e7d7ca290c45348e00493c00de6262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260927 21:38:18.776345 25367 leader_election.cc:302] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4e7d7ca290c45348e00493c00de6262; no voters:
I20260927 21:38:18.776739 25367 leader_election.cc:288] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:18.776827 25377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:18.777444 25377 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [term 1 LEADER]: Becoming Leader. State: Replica: a4e7d7ca290c45348e00493c00de6262, State: Running, Role: LEADER
I20260927 21:38:18.777979 25377 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e7d7ca290c45348e00493c00de6262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260927 21:38:18.778584 25367 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:18.780387 25379 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4e7d7ca290c45348e00493c00de6262. Latest consensus state: current_term: 1 leader_uuid: "a4e7d7ca290c45348e00493c00de6262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e7d7ca290c45348e00493c00de6262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } } }
I20260927 21:38:18.780514 25378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4e7d7ca290c45348e00493c00de6262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e7d7ca290c45348e00493c00de6262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } } }
I20260927 21:38:18.780875 25378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:18.781418 25379 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:18.782330 25382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:18.787169 25382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:18.799657 25382 catalog_manager.cc:1347] Generated new cluster ID: 9f836650c2b9437a8b0afa8213074f1e
I20260927 21:38:18.799746 25382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:18.811411 25382 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:18.812947 25382 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:18.819881 25382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4e7d7ca290c45348e00493c00de6262: Generated new TSK 0
I20260927 21:38:18.820660 25382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:18.826574 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25300
W20260927 21:38:18.827536 25362 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.107:54007
W20260927 21:38:18.828215 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:18.828377 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25300
W20260927 21:38:18.844354 25285 connection.cc:535] client connection to 127.0.0.1:33819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33819): Connection reset by peer (error 104)
2026-09-27T21:38:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1748 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:18 dist-test-slave-pqr8 krb5kdc[25398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:18 dist-test-slave-pqr8 krb5kdc[25398](info): set up 2 sockets
Sep 27 21:38:18 dist-test-slave-pqr8 krb5kdc[25398](info): commencing operation
krb5kdc: starting...
W20260927 21:38:19.893903 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:20 dist-test-slave-pqr8 krb5kdc[25398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:38:20.104914 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:20.407992 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:20.408305 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:20.408465 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:20.415300 25411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:20.415433 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:20.415526 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:20.415624 25411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:20.415727 25411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:20.424978 25411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:20.426157 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:20.427364 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:20.438719 25419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:20.438813 25417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:20.438858 25416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:20.439604 25411 server_base.cc:1034] running on GCE node
I20260927 21:38:20.440240 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:20.441414 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:20.442636 25411 hybrid_clock.cc:648] HybridClock initialized: now 1790545100442593 us; error 34 us; skew 500 ppm
I20260927 21:38:20.442941 25411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:20 dist-test-slave-pqr8 krb5kdc[25398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:20.446012 25411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:38:20.447310 25411 webserver.cc:460] Webserver started at http://127.0.0.1:33219/ using document root <none> and password file <none>
I20260927 21:38:20.447911 25411 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:20.448047 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:20.448341 25411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:20.451098 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "756823d98cb14a94b170261d66116f1c"
format_stamp: "Formatted at 2026-09-27 21:38:20 on dist-test-slave-pqr8"
server_key: "37c312c3476c0bf62d7c0c7a737aebaa"
server_key_iv: "5a63c26e86439b211beb10c6e809057e"
server_key_version: "encryptionkey@0"
I20260927 21:38:20.451823 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "756823d98cb14a94b170261d66116f1c"
format_stamp: "Formatted at 2026-09-27 21:38:20 on dist-test-slave-pqr8"
server_key: "37c312c3476c0bf62d7c0c7a737aebaa"
server_key_iv: "5a63c26e86439b211beb10c6e809057e"
server_key_version: "encryptionkey@0"
I20260927 21:38:20.456445 25411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:38:20.459617 25426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:20.460722 25411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 21:38:20.460879 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "756823d98cb14a94b170261d66116f1c"
format_stamp: "Formatted at 2026-09-27 21:38:20 on dist-test-slave-pqr8"
server_key: "37c312c3476c0bf62d7c0c7a737aebaa"
server_key_iv: "5a63c26e86439b211beb10c6e809057e"
server_key_version: "encryptionkey@0"
I20260927 21:38:20.461045 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:20.466064 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:20.469630 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:20.469888 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:20.486622 25411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40401
I20260927 21:38:20.486687 25477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40401 every 8 connection(s)
I20260927 21:38:20.487829 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:20.491150 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:20.492224 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25411
I20260927 21:38:20.492401 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:20.492888 3819 external_mini_cluster.cc:1428] Setting key 1de938e96d4621dc075626505950c180
I20260927 21:38:20.499753 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:20.494458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:42471 (local address 127.0.0.1:40401)
0927 21:38:20.495219 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:20.495230 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:20.495268 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:38:20.496794 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:20.496815 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:20.497118 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:20.497411 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:20.498397 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:20.498403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:20.498414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:20.498432 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:20.498481 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:20.498498 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:20.498650 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:20.498762 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 21:38:20.498956 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":344,"threads_started":1}
I20260927 21:38:20.503260 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:20.503824 25478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c: Bootstrap starting.
I20260927 21:38:20.507412 25478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:20.508589 25478 log.cc:824] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:20.510627 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:20.508196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:48261 (local address 127.0.0.1:40401)
0927 21:38:20.508367 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:20.508376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:20.508809 (+ 433us) server_negotiation.cc:407] Connection header received
0927 21:38:20.508926 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:20.508932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:20.508986 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:20.509114 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:20.510017 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:20.510024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:20.510031 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:20.510052 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:20.510090 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:20.510107 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:20.510211 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:20.510305 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:38:20.510373 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:38:20.511737 25478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c: No bootstrap required, opened a new log
I20260927 21:38:20.518448 25478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756823d98cb14a94b170261d66116f1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260927 21:38:20.518775 25478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:20.518874 25478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756823d98cb14a94b170261d66116f1c, State: Initialized, Role: FOLLOWER
I20260927 21:38:20.519419 25478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756823d98cb14a94b170261d66116f1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260927 21:38:20.519649 25478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:20.519778 25478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:20.519934 25478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:20.521909 25478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756823d98cb14a94b170261d66116f1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260927 21:38:20.522338 25478 leader_election.cc:302] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 756823d98cb14a94b170261d66116f1c; no voters:
I20260927 21:38:20.522831 25478 leader_election.cc:288] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:20.522945 25488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:20.523574 25488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [term 1 LEADER]: Becoming Leader. State: Replica: 756823d98cb14a94b170261d66116f1c, State: Running, Role: LEADER
I20260927 21:38:20.524101 25488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756823d98cb14a94b170261d66116f1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260927 21:38:20.524722 25478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:20.526551 25490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 756823d98cb14a94b170261d66116f1c. Latest consensus state: current_term: 1 leader_uuid: "756823d98cb14a94b170261d66116f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756823d98cb14a94b170261d66116f1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } } }
I20260927 21:38:20.526962 25490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:20.527511 25489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "756823d98cb14a94b170261d66116f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756823d98cb14a94b170261d66116f1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } } }
I20260927 21:38:20.527778 25489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:20.529407 25497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:20.533252 25497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:20.540932 25497 catalog_manager.cc:1347] Generated new cluster ID: 2dc973728c024a72bfb7ca57232a5432
I20260927 21:38:20.541033 25497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:20.572090 25497 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:20.572459 25497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:20.579717 25497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 756823d98cb14a94b170261d66116f1c: Generated new TSK 0
I20260927 21:38:20.580400 25497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:20.584594 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25411
W20260927 21:38:20.585521 25473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.107:42471
W20260927 21:38:20.586143 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:20.586268 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25411
2026-09-27T21:38:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1758 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:20 dist-test-slave-pqr8 krb5kdc[25509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:20 dist-test-slave-pqr8 krb5kdc[25509](info): set up 2 sockets
Sep 27 21:38:20 dist-test-slave-pqr8 krb5kdc[25509](info): commencing operation
krb5kdc: starting...
W20260927 21:38:21.649839 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:21 dist-test-slave-pqr8 krb5kdc[25509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:21.862974 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:45625
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:22.162739 25522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:22.163015 25522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:22.163167 25522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:22.169898 25522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:22.170020 25522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:22.170112 25522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:22.170287 25522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:22.170433 25522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:22.179281 25522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:45625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:22.180361 25522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:22.181484 25522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:22.192226 25527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:22.192250 25528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:22.192471 25530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:22.192991 25522 server_base.cc:1034] running on GCE node
I20260927 21:38:22.193729 25522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:22.194882 25522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:22.196102 25522 hybrid_clock.cc:648] HybridClock initialized: now 1790545102196063 us; error 30 us; skew 500 ppm
I20260927 21:38:22.196421 25522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:22 dist-test-slave-pqr8 krb5kdc[25509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:22.203357 25522 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:22.204602 25522 webserver.cc:460] Webserver started at http://127.3.186.254:44011/ using document root <none> and password file <none>
I20260927 21:38:22.205132 25522 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:22.205253 25522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:22.205484 25522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:22.208158 25522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "cfa31bf9000345e2869d65425c2cfa00"
format_stamp: "Formatted at 2026-09-27 21:38:22 on dist-test-slave-pqr8"
server_key: "4be698341da2bb2d0b5135b7b1ad26ad"
server_key_iv: "64e1f1870523d4acb5ab31cae9e534d3"
server_key_version: "encryptionkey@0"
I20260927 21:38:22.208863 25522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "cfa31bf9000345e2869d65425c2cfa00"
format_stamp: "Formatted at 2026-09-27 21:38:22 on dist-test-slave-pqr8"
server_key: "4be698341da2bb2d0b5135b7b1ad26ad"
server_key_iv: "64e1f1870523d4acb5ab31cae9e534d3"
server_key_version: "encryptionkey@0"
I20260927 21:38:22.213399 25522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 21:38:22.216460 25537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:22.217738 25522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:38:22.217871 25522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "cfa31bf9000345e2869d65425c2cfa00"
format_stamp: "Formatted at 2026-09-27 21:38:22 on dist-test-slave-pqr8"
server_key: "4be698341da2bb2d0b5135b7b1ad26ad"
server_key_iv: "64e1f1870523d4acb5ab31cae9e534d3"
server_key_version: "encryptionkey@0"
I20260927 21:38:22.218017 25522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:22.244906 25522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:22.248538 25522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:22.248757 25522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:22.266171 25522 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:45625
I20260927 21:38:22.266175 25588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:45625 every 8 connection(s)
I20260927 21:38:22.267953 25522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:22.269851 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25522
I20260927 21:38:22.270020 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:22.270447 3819 external_mini_cluster.cc:1428] Setting key 61ccb21e37889107217b1f9d9b870c87
I20260927 21:38:22.271942 25589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:38:22 dist-test-slave-pqr8 krb5kdc[25509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:22.285079 25589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00: Bootstrap starting.
I20260927 21:38:22.288640 25589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:22.289846 25589 log.cc:824] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:22.292761 25589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00: No bootstrap required, opened a new log
I20260927 21:38:22.293294 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:22.272067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43269 (local address 127.3.186.254:45625)
0927 21:38:22.272802 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:22.272818 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:22.272865 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:38:22.274894 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:22.274919 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:22.275395 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:22.275865 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:22.277086 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:22.279100 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:22.280302 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:22.281342 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:22.281549 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:22.285795 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:22.285802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:22.285814 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:22.285846 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:22.289259 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:22.290183 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:22.290189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:22.290194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:22.290403 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:22.290864 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:22.290870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:22.290876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:22.291391 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:22.291621 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:22.292112 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 21:38:22.292366 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":302,"threads_started":1}
I20260927 21:38:22.296797 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:22.301931 25589 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa31bf9000345e2869d65425c2cfa00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45625 } }
I20260927 21:38:22.302255 25589 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:22.302373 25589 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa31bf9000345e2869d65425c2cfa00, State: Initialized, Role: FOLLOWER
I20260927 21:38:22.302969 25589 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa31bf9000345e2869d65425c2cfa00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45625 } }
I20260927 21:38:22.303190 25589 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:22.303341 25589 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:22.303510 25589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:22.306151 25589 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa31bf9000345e2869d65425c2cfa00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45625 } }
I20260927 21:38:22.306632 25589 leader_election.cc:302] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfa31bf9000345e2869d65425c2cfa00; no voters:
I20260927 21:38:22.307225 25589 leader_election.cc:288] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:22.307333 25599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:22.309267 25589 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:22.310003 25599 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [term 1 LEADER]: Becoming Leader. State: Replica: cfa31bf9000345e2869d65425c2cfa00, State: Running, Role: LEADER
I20260927 21:38:22.310519 25599 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa31bf9000345e2869d65425c2cfa00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45625 } }
I20260927 21:38:22.313539 25600 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfa31bf9000345e2869d65425c2cfa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa31bf9000345e2869d65425c2cfa00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45625 } } }
I20260927 21:38:22.313570 25601 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa31bf9000345e2869d65425c2cfa00. Latest consensus state: current_term: 1 leader_uuid: "cfa31bf9000345e2869d65425c2cfa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa31bf9000345e2869d65425c2cfa00" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 45625 } } }
I20260927 21:38:22.313959 25601 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:22.313956 25600 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:22.320338 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:22.301633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57027 (local address 127.3.186.254:45625)
0927 21:38:22.301809 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:22.301814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:22.302299 (+ 485us) server_negotiation.cc:407] Connection header received
0927 21:38:22.302466 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:22.302472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:22.302562 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:22.302679 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:22.304207 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:22.305350 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:22.306635 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:22.307382 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:22.307517 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:22.310849 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:22.310857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:22.310865 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:22.310906 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:22.315175 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:22.316371 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:22.316380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:22.316386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:22.316583 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:22.317341 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:22.317348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:22.317354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:22.317833 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:22.319294 (+ 1461us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:22.319517 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:38:22.319779 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260927 21:38:22.322098 25607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:22.326170 25607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:22.333860 25607 catalog_manager.cc:1347] Generated new cluster ID: 2f94fff049764a59bf470b6fa9a2d04d
I20260927 21:38:22.333954 25607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:22.347159 25607 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:22.348259 25607 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:22.353960 25607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfa31bf9000345e2869d65425c2cfa00: Generated new TSK 0
I20260927 21:38:22.354576 25607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:22.363209 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25522
I20260927 21:38:22.364297 25584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:22.448920 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25522
W20260927 21:38:22.465709 25507 connection.cc:535] client connection to 127.3.186.254:45625 recv error: Network error: failed to read from TLS socket (remote: 127.3.186.254:45625): Connection reset by peer (error 104)
2026-09-27T21:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:22 dist-test-slave-pqr8 krb5kdc[25621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:22 dist-test-slave-pqr8 krb5kdc[25621](info): set up 2 sockets
Sep 27 21:38:22 dist-test-slave-pqr8 krb5kdc[25621](info): commencing operation
krb5kdc: starting...
W20260927 21:38:23.505585 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:23 dist-test-slave-pqr8 krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:23.714043 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:35981
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:24.016551 25634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:24.016821 25634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:24.016997 25634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:24.023481 25634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:24.023627 25634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:24.023736 25634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:24.023838 25634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:24.023922 25634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:24.032665 25634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:35981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:35981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:24.033721 25634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:24.034871 25634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:24.046032 25640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:24.046118 25642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:24.046088 25639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:24.046908 25634 server_base.cc:1034] running on GCE node
I20260927 21:38:24.047681 25634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:24.048943 25634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:24.050168 25634 hybrid_clock.cc:648] HybridClock initialized: now 1790545104050118 us; error 41 us; skew 500 ppm
I20260927 21:38:24.050489 25634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:24 dist-test-slave-pqr8 krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:24.053376 25634 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:24.054745 25634 webserver.cc:460] Webserver started at http://127.3.186.254:43721/ using document root <none> and password file <none>
I20260927 21:38:24.055377 25634 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:24.055521 25634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:24.055805 25634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:24.058568 25634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "bbf454c7ece2404fa2a4428e75341cde"
format_stamp: "Formatted at 2026-09-27 21:38:24 on dist-test-slave-pqr8"
server_key: "c1c3f99b2645f68b3b3c9a6ea6e5823d"
server_key_iv: "71a585e241e1be990f2dd7f3e2f3ac1d"
server_key_version: "encryptionkey@0"
I20260927 21:38:24.059304 25634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "bbf454c7ece2404fa2a4428e75341cde"
format_stamp: "Formatted at 2026-09-27 21:38:24 on dist-test-slave-pqr8"
server_key: "c1c3f99b2645f68b3b3c9a6ea6e5823d"
server_key_iv: "71a585e241e1be990f2dd7f3e2f3ac1d"
server_key_version: "encryptionkey@0"
I20260927 21:38:24.063895 25634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:38:24.067028 25649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:24.068190 25634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:38:24.068373 25634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "bbf454c7ece2404fa2a4428e75341cde"
format_stamp: "Formatted at 2026-09-27 21:38:24 on dist-test-slave-pqr8"
server_key: "c1c3f99b2645f68b3b3c9a6ea6e5823d"
server_key_iv: "71a585e241e1be990f2dd7f3e2f3ac1d"
server_key_version: "encryptionkey@0"
I20260927 21:38:24.068553 25634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:24.081463 25634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:24.084805 25634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:24.085057 25634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:24.102236 25634 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:35981
I20260927 21:38:24.102275 25700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:35981 every 8 connection(s)
I20260927 21:38:24.103392 25634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:24.106626 25701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:24.109987 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25634
I20260927 21:38:24.110153 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:24.110663 3819 external_mini_cluster.cc:1428] Setting key ebe9d3b10c6fdca11116b0448ccfa817
I20260927 21:38:24.117455 25701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde: Bootstrap starting.
I20260927 21:38:24.120296 25701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:24.121268 25701 log.cc:824] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:24.122371 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:24.112270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43933 (local address 127.3.186.254:35981)
0927 21:38:24.112909 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:24.112920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:24.112968 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:38:24.114525 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:24.114544 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:24.114893 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:24.115243 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:24.115749 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:24.116931 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:24.118290 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:24.118936 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:24.119146 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:24.120270 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:24.120277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:24.120291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:24.120798 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:24.120859 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:24.120898 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:24.121061 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:24.121330 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 21:38:24.121492 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":281,"threads_started":1}
I20260927 21:38:24.123832 25701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde: No bootstrap required, opened a new log
I20260927 21:38:24.125809 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:24.131628 25701 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf454c7ece2404fa2a4428e75341cde" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35981 } }
I20260927 21:38:24.131989 25701 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:24.132081 25701 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf454c7ece2404fa2a4428e75341cde, State: Initialized, Role: FOLLOWER
I20260927 21:38:24.132575 25701 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf454c7ece2404fa2a4428e75341cde" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35981 } }
I20260927 21:38:24.132784 25701 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:24.132931 25701 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:24.133070 25701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:24.135818 25701 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf454c7ece2404fa2a4428e75341cde" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35981 } }
I20260927 21:38:24.136025 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:24.129708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36351 (local address 127.3.186.254:35981)
0927 21:38:24.129867 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:24.129878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:24.130395 (+ 517us) server_negotiation.cc:407] Connection header received
0927 21:38:24.130523 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:24.130528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:24.130593 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:24.130693 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:24.131070 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:24.132125 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:24.133621 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:24.134286 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:24.134407 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:24.135279 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:24.135284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:24.135288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:24.135304 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:24.135358 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:24.135375 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:24.135469 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:24.135599 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:38:24.135645 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:38:24.136366 25701 leader_election.cc:302] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf454c7ece2404fa2a4428e75341cde; no voters:
I20260927 21:38:24.136905 25701 leader_election.cc:288] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:24.137066 25711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:24.137892 25711 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [term 1 LEADER]: Becoming Leader. State: Replica: bbf454c7ece2404fa2a4428e75341cde, State: Running, Role: LEADER
I20260927 21:38:24.138353 25711 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf454c7ece2404fa2a4428e75341cde" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35981 } }
I20260927 21:38:24.138788 25701 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:24.140859 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf454c7ece2404fa2a4428e75341cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf454c7ece2404fa2a4428e75341cde" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35981 } } }
I20260927 21:38:24.140870 25713 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf454c7ece2404fa2a4428e75341cde. Latest consensus state: current_term: 1 leader_uuid: "bbf454c7ece2404fa2a4428e75341cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf454c7ece2404fa2a4428e75341cde" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 35981 } } }
I20260927 21:38:24.141256 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:24.141263 25713 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:24.146656 25717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:24.150970 25717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:24.158706 25717 catalog_manager.cc:1347] Generated new cluster ID: 7b46edcd5b6c4552a792e707e431c134
I20260927 21:38:24.158802 25717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:24.173026 25717 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:24.174662 25717 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:24.184408 25717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf454c7ece2404fa2a4428e75341cde: Generated new TSK 0
I20260927 21:38:24.185165 25717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:24.190959 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25634
W20260927 21:38:24.191869 25696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43933
W20260927 21:38:24.192472 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25634: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:24.192580 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25634
W20260927 21:38:24.208696 25619 connection.cc:535] client connection to 127.3.186.254:35981 recv error: Network error: failed to read from TLS socket (remote: 127.3.186.254:35981): Connection reset by peer (error 104)
2026-09-27T21:38:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:24 dist-test-slave-pqr8 krb5kdc[25732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:24 dist-test-slave-pqr8 krb5kdc[25732](info): set up 2 sockets
Sep 27 21:38:24 dist-test-slave-pqr8 krb5kdc[25732](info): commencing operation
krb5kdc: starting...
W20260927 21:38:25.237919 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:25 dist-test-slave-pqr8 krb5kdc[25732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:25.457995 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:33985
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:33985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:25.767665 25745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:25.767951 25745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:25.768101 25745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:25.774752 25745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:25.774876 25745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:25.774981 25745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:25.775082 25745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:25.775164 25745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:25.786635 25745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:33985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:25.788244 25745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:25.789829 25745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:25.800889 25753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:25.801026 25751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:25.801056 25750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:25.801898 25745 server_base.cc:1034] running on GCE node
I20260927 21:38:25.802558 25745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:25.803753 25745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:25.804961 25745 hybrid_clock.cc:648] HybridClock initialized: now 1790545105804929 us; error 27 us; skew 500 ppm
I20260927 21:38:25.805272 25745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:25 dist-test-slave-pqr8 krb5kdc[25732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:25.808252 25745 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:25.809688 25745 webserver.cc:460] Webserver started at http://127.3.186.254:41081/ using document root <none> and password file <none>
I20260927 21:38:25.810307 25745 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:25.810451 25745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:25.810731 25745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:25.813484 25745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "abe8028a04c64909ab2e895c8253d36d"
format_stamp: "Formatted at 2026-09-27 21:38:25 on dist-test-slave-pqr8"
server_key: "9a5ce86c774af4cc137ec6904a69ea7e"
server_key_iv: "36151d9e1e940a29b3e525db0202b747"
server_key_version: "encryptionkey@0"
I20260927 21:38:25.814301 25745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "abe8028a04c64909ab2e895c8253d36d"
format_stamp: "Formatted at 2026-09-27 21:38:25 on dist-test-slave-pqr8"
server_key: "9a5ce86c774af4cc137ec6904a69ea7e"
server_key_iv: "36151d9e1e940a29b3e525db0202b747"
server_key_version: "encryptionkey@0"
I20260927 21:38:25.819054 25745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:38:25.822355 25760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:25.823567 25745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:38:25.823763 25745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "abe8028a04c64909ab2e895c8253d36d"
format_stamp: "Formatted at 2026-09-27 21:38:25 on dist-test-slave-pqr8"
server_key: "9a5ce86c774af4cc137ec6904a69ea7e"
server_key_iv: "36151d9e1e940a29b3e525db0202b747"
server_key_version: "encryptionkey@0"
I20260927 21:38:25.823951 25745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:25.829178 25745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:25.832902 25745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:25.833192 25745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:25.850945 25745 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:33985
I20260927 21:38:25.850981 25811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:33985 every 8 connection(s)
I20260927 21:38:25.852146 25745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:25.852649 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25745
I20260927 21:38:25.852859 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:25.853456 3819 external_mini_cluster.cc:1428] Setting key b076c2465d60dee63954ecba6043c054
I20260927 21:38:25.856626 25812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:25.861574 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:25.855050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47449 (local address 127.3.186.254:33985)
0927 21:38:25.855853 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:25.855867 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:25.855913 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:38:25.858076 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:25.858100 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:25.858493 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:25.858878 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:25.859791 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:25.859798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:25.859813 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:25.859834 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:25.859920 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:25.859943 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:25.860171 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:25.860301 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:38:25.860456 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":344,"threads_started":1}
I20260927 21:38:25.864899 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:25.868772 25812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d: Bootstrap starting.
I20260927 21:38:25.871330 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:25.868962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34369 (local address 127.3.186.254:33985)
0927 21:38:25.869162 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:25.869171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:25.869562 (+ 391us) server_negotiation.cc:407] Connection header received
0927 21:38:25.869691 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:25.869694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:25.869750 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:25.869822 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:25.870699 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:25.870703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:25.870706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:25.870733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:25.870768 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:25.870784 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:25.870867 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:25.871007 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:38:25.871048 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:38:25.872474 25812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:25.873379 25812 log.cc:824] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:25.875727 25812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d: No bootstrap required, opened a new log
I20260927 21:38:25.882858 25812 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe8028a04c64909ab2e895c8253d36d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33985 } }
I20260927 21:38:25.883172 25812 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:25.883281 25812 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe8028a04c64909ab2e895c8253d36d, State: Initialized, Role: FOLLOWER
I20260927 21:38:25.883836 25812 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe8028a04c64909ab2e895c8253d36d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33985 } }
I20260927 21:38:25.884052 25812 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:25.884195 25812 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:25.884368 25812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:25.886653 25812 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe8028a04c64909ab2e895c8253d36d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33985 } }
I20260927 21:38:25.887112 25812 leader_election.cc:302] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe8028a04c64909ab2e895c8253d36d; no voters:
I20260927 21:38:25.887600 25812 leader_election.cc:288] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:25.887698 25822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:25.888410 25822 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [term 1 LEADER]: Becoming Leader. State: Replica: abe8028a04c64909ab2e895c8253d36d, State: Running, Role: LEADER
I20260927 21:38:25.888940 25822 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe8028a04c64909ab2e895c8253d36d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33985 } }
I20260927 21:38:25.889532 25812 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:25.891677 25824 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe8028a04c64909ab2e895c8253d36d. Latest consensus state: current_term: 1 leader_uuid: "abe8028a04c64909ab2e895c8253d36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe8028a04c64909ab2e895c8253d36d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33985 } } }
I20260927 21:38:25.892098 25824 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:25.892303 25823 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe8028a04c64909ab2e895c8253d36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe8028a04c64909ab2e895c8253d36d" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 33985 } } }
I20260927 21:38:25.892637 25823 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:25.894256 25830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:25.898967 25830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:25.906100 25830 catalog_manager.cc:1347] Generated new cluster ID: 3d98e0bee9264985a00252d3897114f0
I20260927 21:38:25.906195 25830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:25.931200 25830 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:25.931663 25830 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:25.937825 25830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe8028a04c64909ab2e895c8253d36d: Generated new TSK 0
I20260927 21:38:25.938606 25830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:25.950129 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25745
W20260927 21:38:25.951015 25807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47449
W20260927 21:38:25.951604 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:25.951762 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25745
2026-09-27T21:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:26 dist-test-slave-pqr8 krb5kdc[25843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:26 dist-test-slave-pqr8 krb5kdc[25843](info): set up 2 sockets
Sep 27 21:38:26 dist-test-slave-pqr8 krb5kdc[25843](info): commencing operation
krb5kdc: starting...
W20260927 21:38:27.013528 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:27 dist-test-slave-pqr8 krb5kdc[25843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:27.218691 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41321
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:27.520394 25856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:27.520720 25856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:27.520884 25856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:27.527482 25856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:27.527613 25856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:27.527722 25856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:27.527810 25856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:27.527873 25856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:27.536600 25856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:41321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:27.537654 25856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:27.538844 25856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:27.549629 25862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:27.549691 25864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:27.549687 25861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:27.550552 25856 server_base.cc:1034] running on GCE node
I20260927 21:38:27.551229 25856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:27.552410 25856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:27.553603 25856 hybrid_clock.cc:648] HybridClock initialized: now 1790545107553557 us; error 35 us; skew 500 ppm
I20260927 21:38:27.553974 25856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:27 dist-test-slave-pqr8 krb5kdc[25843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545107, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:27.562336 25856 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:27.563624 25856 webserver.cc:460] Webserver started at http://127.3.186.254:42567/ using document root <none> and password file <none>
I20260927 21:38:27.564215 25856 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:27.564363 25856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:27.564642 25856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:27.567338 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "7a887b6f7d3d46c78840c8e4ad7d5654"
format_stamp: "Formatted at 2026-09-27 21:38:27 on dist-test-slave-pqr8"
server_key: "3863c52d28979253eb92fbc90bd5a3d1"
server_key_iv: "b23c72cb2016b47391f63039f678398a"
server_key_version: "encryptionkey@0"
I20260927 21:38:27.568075 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "7a887b6f7d3d46c78840c8e4ad7d5654"
format_stamp: "Formatted at 2026-09-27 21:38:27 on dist-test-slave-pqr8"
server_key: "3863c52d28979253eb92fbc90bd5a3d1"
server_key_iv: "b23c72cb2016b47391f63039f678398a"
server_key_version: "encryptionkey@0"
I20260927 21:38:27.572700 25856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 21:38:27.575656 25871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:27.576831 25856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:38:27.577004 25856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "7a887b6f7d3d46c78840c8e4ad7d5654"
format_stamp: "Formatted at 2026-09-27 21:38:27 on dist-test-slave-pqr8"
server_key: "3863c52d28979253eb92fbc90bd5a3d1"
server_key_iv: "b23c72cb2016b47391f63039f678398a"
server_key_version: "encryptionkey@0"
I20260927 21:38:27.577178 25856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:27.596810 25856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:27.600342 25856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:27.600625 25856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:27.617992 25856 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:41321
I20260927 21:38:27.618021 25922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:41321 every 8 connection(s)
I20260927 21:38:27.619171 25856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:27.622480 25923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:27.625738 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25856
I20260927 21:38:27.625912 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:27.626345 3819 external_mini_cluster.cc:1428] Setting key 1249ef0702bdb879c1b8d1e321ff89fb
I20260927 21:38:27.634776 25923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654: Bootstrap starting.
I20260927 21:38:27.637965 25923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:27.638907 25923 log.cc:824] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:38:27 dist-test-slave-pqr8 krb5kdc[25843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:27.641917 25923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654: No bootstrap required, opened a new log
I20260927 21:38:27.648591 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:27.627998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54805 (local address 127.3.186.254:41321)
0927 21:38:27.628761 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:27.628773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:27.628820 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:38:27.630842 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:27.630870 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:27.631356 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:27.631742 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:27.633038 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:27.634958 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:27.636168 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:27.637187 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:27.637357 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:27.641869 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:27.641875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:27.641887 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:27.641926 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:27.644848 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:27.645670 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:27.645676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:27.645681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:27.645858 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:27.646263 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:27.646266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:27.646268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:27.646709 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:27.646946 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:27.647356 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 21:38:27.647651 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":340,"threads_started":1}
I20260927 21:38:27.650264 25923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41321 } }
I20260927 21:38:27.650636 25923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:27.650753 25923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a887b6f7d3d46c78840c8e4ad7d5654, State: Initialized, Role: FOLLOWER
I20260927 21:38:27.651324 25923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41321 } }
I20260927 21:38:27.651602 25923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:27.651762 25923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:27.651932 25923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:27.652305 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:27.654093 25923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41321 } }
I20260927 21:38:27.654526 25923 leader_election.cc:302] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a887b6f7d3d46c78840c8e4ad7d5654; no voters:
I20260927 21:38:27.655252 25923 leader_election.cc:288] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:27.655333 25930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:27.657598 25923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:27.658017 25930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [term 1 LEADER]: Becoming Leader. State: Replica: 7a887b6f7d3d46c78840c8e4ad7d5654, State: Running, Role: LEADER
I20260927 21:38:27.658632 25930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41321 } }
I20260927 21:38:27.661562 25933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a887b6f7d3d46c78840c8e4ad7d5654. Latest consensus state: current_term: 1 leader_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41321 } } }
I20260927 21:38:27.661980 25933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:27.662509 25934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a887b6f7d3d46c78840c8e4ad7d5654" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 41321 } } }
I20260927 21:38:27.662855 25934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:27.671114 25938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:27.675843 25938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:27.689553 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:27.668445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33015 (local address 127.3.186.254:41321)
0927 21:38:27.668596 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:27.668601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:27.669764 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 21:38:27.669938 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:27.669946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:27.670024 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:27.670130 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:27.672127 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:27.673607 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:27.675942 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:27.676822 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:27.676977 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:27.680351 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:27.680361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:27.680369 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:27.680415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:27.684776 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:27.685823 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:27.685832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:27.685838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:27.686057 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:27.686594 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:27.686602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:27.686607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:27.687099 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:27.687417 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:27.688738 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0927 21:38:27.689007 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:38:27.694406 25938 catalog_manager.cc:1347] Generated new cluster ID: c5f0de36ec9245dbbc9453199d315619
I20260927 21:38:27.694497 25938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:27.712859 25938 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:27.714010 25938 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:27.722517 25938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a887b6f7d3d46c78840c8e4ad7d5654: Generated new TSK 0
I20260927 21:38:27.723363 25938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:27.730955 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25856
I20260927 21:38:27.731890 25918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:27.827026 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25856
2026-09-27T21:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:27 dist-test-slave-pqr8 krb5kdc[25956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:27 dist-test-slave-pqr8 krb5kdc[25956](info): set up 2 sockets
Sep 27 21:38:27 dist-test-slave-pqr8 krb5kdc[25956](info): commencing operation
krb5kdc: starting...
W20260927 21:38:28.881978 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.007s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:29 dist-test-slave-pqr8 krb5kdc[25956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:29.092799 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:38921
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:38921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:29.402412 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:29.402735 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:29.402911 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:29.409960 25969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:29.410092 25969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:29.410192 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:29.410286 25969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:29.410351 25969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:29.419088 25969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:38921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:38921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:29.420195 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:29.421375 25969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:29.432736 25974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:29.432816 25975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:29.432843 25977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:29.433758 25969 server_base.cc:1034] running on GCE node
I20260927 21:38:29.434628 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:29.436136 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:29.437354 25969 hybrid_clock.cc:648] HybridClock initialized: now 1790545109437315 us; error 59 us; skew 500 ppm
I20260927 21:38:29.437744 25969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:29 dist-test-slave-pqr8 krb5kdc[25956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545109, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:29.440547 25969 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:29.441838 25969 webserver.cc:460] Webserver started at http://127.3.186.254:39083/ using document root <none> and password file <none>
I20260927 21:38:29.442459 25969 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:29.442598 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:29.442873 25969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:29.445605 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "518d4fa5ef904303ac3eb591751ed68f"
format_stamp: "Formatted at 2026-09-27 21:38:29 on dist-test-slave-pqr8"
server_key: "d12f91025061e7512d3f2160e3ea13ba"
server_key_iv: "0d3e2942d6871bc93fc6689ee203390a"
server_key_version: "encryptionkey@0"
I20260927 21:38:29.446378 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "518d4fa5ef904303ac3eb591751ed68f"
format_stamp: "Formatted at 2026-09-27 21:38:29 on dist-test-slave-pqr8"
server_key: "d12f91025061e7512d3f2160e3ea13ba"
server_key_iv: "0d3e2942d6871bc93fc6689ee203390a"
server_key_version: "encryptionkey@0"
I20260927 21:38:29.450917 25969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:38:29.454056 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:29.455127 25969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:29.455293 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "518d4fa5ef904303ac3eb591751ed68f"
format_stamp: "Formatted at 2026-09-27 21:38:29 on dist-test-slave-pqr8"
server_key: "d12f91025061e7512d3f2160e3ea13ba"
server_key_iv: "0d3e2942d6871bc93fc6689ee203390a"
server_key_version: "encryptionkey@0"
I20260927 21:38:29.455466 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:29.493394 25969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:29.496893 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:29.497161 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:29.514329 25969 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:38921
I20260927 21:38:29.514364 26035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:38921 every 8 connection(s)
I20260927 21:38:29.515497 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:29.518883 26036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:29.521906 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 25969
I20260927 21:38:29.522074 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:29.522488 3819 external_mini_cluster.cc:1428] Setting key fb05bb287a4bcd7b07150b4ac9c03990
I20260927 21:38:29.529999 26036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f: Bootstrap starting.
I20260927 21:38:29.533233 26036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:29.534343 26036 log.cc:824] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:29.534785 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:29.524092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35243 (local address 127.3.186.254:38921)
0927 21:38:29.524821 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:29.524831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:29.524867 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:38:29.526460 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:29.526479 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:29.526848 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:29.527154 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:29.527688 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:29.528921 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:29.530685 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:29.531286 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:29.531410 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:29.532639 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:29.532646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:29.532657 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:29.533114 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:29.533181 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:29.533204 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:29.533283 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:29.533586 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:38:29.533855 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":314,"threads_started":1}
I20260927 21:38:29.537007 26036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f: No bootstrap required, opened a new log
I20260927 21:38:29.538018 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:29.545140 26036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518d4fa5ef904303ac3eb591751ed68f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38921 } }
I20260927 21:38:29.545532 26036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:29.545660 26036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518d4fa5ef904303ac3eb591751ed68f, State: Initialized, Role: FOLLOWER
I20260927 21:38:29.546208 26036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518d4fa5ef904303ac3eb591751ed68f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38921 } }
I20260927 21:38:29.546479 26036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:29.546617 26036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:29.546777 26036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:29.547991 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:29.541964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39101 (local address 127.3.186.254:38921)
0927 21:38:29.542185 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:29.542193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:29.542632 (+ 439us) server_negotiation.cc:407] Connection header received
0927 21:38:29.542786 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:29.542790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:29.542839 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:29.542917 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:29.543459 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:29.544124 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:29.545887 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:29.546284 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:29.546372 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:29.547404 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:29.547408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:29.547411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:29.547424 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:29.547455 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:29.547467 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:29.547531 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:29.547652 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:38:29.547707 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:38:29.549182 26036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518d4fa5ef904303ac3eb591751ed68f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38921 } }
I20260927 21:38:29.549705 26036 leader_election.cc:302] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518d4fa5ef904303ac3eb591751ed68f; no voters:
I20260927 21:38:29.550180 26036 leader_election.cc:288] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:29.550310 26046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:29.550981 26046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [term 1 LEADER]: Becoming Leader. State: Replica: 518d4fa5ef904303ac3eb591751ed68f, State: Running, Role: LEADER
I20260927 21:38:29.551487 26046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518d4fa5ef904303ac3eb591751ed68f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38921 } }
I20260927 21:38:29.551997 26036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:29.554155 26048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518d4fa5ef904303ac3eb591751ed68f. Latest consensus state: current_term: 1 leader_uuid: "518d4fa5ef904303ac3eb591751ed68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518d4fa5ef904303ac3eb591751ed68f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38921 } } }
I20260927 21:38:29.554593 26048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:29.555330 26047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518d4fa5ef904303ac3eb591751ed68f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518d4fa5ef904303ac3eb591751ed68f" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38921 } } }
I20260927 21:38:29.555619 26047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:29.555828 26055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:29.561664 26055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:29.569222 26055 catalog_manager.cc:1347] Generated new cluster ID: b3114fb6d47844749b778424ff6957d6
I20260927 21:38:29.569319 26055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:29.597543 26055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:29.598878 26055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:29.606542 26055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518d4fa5ef904303ac3eb591751ed68f: Generated new TSK 0
I20260927 21:38:29.607244 26055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:29.615981 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25969
W20260927 21:38:29.616909 26031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35243
W20260927 21:38:29.617585 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 25969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:29.617758 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 25969
2026-09-27T21:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1790 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:29 dist-test-slave-pqr8 krb5kdc[26067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:29 dist-test-slave-pqr8 krb5kdc[26067](info): set up 2 sockets
Sep 27 21:38:29 dist-test-slave-pqr8 krb5kdc[26067](info): commencing operation
krb5kdc: starting...
W20260927 21:38:30.674080 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:30 dist-test-slave-pqr8 krb5kdc[26067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:30.884018 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:38275
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:31.181054 26080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:31.181288 26080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:31.181450 26080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:31.188144 26080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:31.188265 26080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:31.188364 26080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:31.188467 26080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:31.188548 26080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:31.197319 26080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:31.198336 26080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:31.199532 26080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:31.210278 26085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:31.210350 26088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:31.210399 26086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:31.211335 26080 server_base.cc:1034] running on GCE node
I20260927 21:38:31.211993 26080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:31.213198 26080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:31.214417 26080 hybrid_clock.cc:648] HybridClock initialized: now 1790545111214391 us; error 44 us; skew 500 ppm
I20260927 21:38:31.214761 26080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:31 dist-test-slave-pqr8 krb5kdc[26067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545111, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:31.217598 26080 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:31.218962 26080 webserver.cc:460] Webserver started at http://127.3.186.254:41121/ using document root <none> and password file <none>
I20260927 21:38:31.219621 26080 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:31.219765 26080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:31.220031 26080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:31.222819 26080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "c6d0b9d94a384c4c83eb668a2710f440"
format_stamp: "Formatted at 2026-09-27 21:38:31 on dist-test-slave-pqr8"
server_key: "765406896a9c0e2cf10f59da2ba8a335"
server_key_iv: "ffcc21a8d20d617736a8373d5e6eb1e8"
server_key_version: "encryptionkey@0"
I20260927 21:38:31.223512 26080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "c6d0b9d94a384c4c83eb668a2710f440"
format_stamp: "Formatted at 2026-09-27 21:38:31 on dist-test-slave-pqr8"
server_key: "765406896a9c0e2cf10f59da2ba8a335"
server_key_iv: "ffcc21a8d20d617736a8373d5e6eb1e8"
server_key_version: "encryptionkey@0"
I20260927 21:38:31.228036 26080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 21:38:31.231137 26095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:31.232228 26080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:38:31.232386 26080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "c6d0b9d94a384c4c83eb668a2710f440"
format_stamp: "Formatted at 2026-09-27 21:38:31 on dist-test-slave-pqr8"
server_key: "765406896a9c0e2cf10f59da2ba8a335"
server_key_iv: "ffcc21a8d20d617736a8373d5e6eb1e8"
server_key_version: "encryptionkey@0"
I20260927 21:38:31.232560 26080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:31.237545 26080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:31.240978 26080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:31.241230 26080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:31.258304 26080 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:38275
I20260927 21:38:31.258371 26146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:38275 every 8 connection(s)
I20260927 21:38:31.259495 26080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:31.260295 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26080
I20260927 21:38:31.260485 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:31.260854 3819 external_mini_cluster.cc:1428] Setting key 5c7e2ca340b62406db2573f00182891f
I20260927 21:38:31.263317 26147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:31.267632 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:31.262316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41385 (local address 127.3.186.254:38275)
0927 21:38:31.263050 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:31.263060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:31.263096 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:38:31.264717 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:31.264737 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:31.265024 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:31.265309 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:31.266290 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:31.266295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:31.266307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:31.266327 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:31.266388 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:31.266404 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:31.266551 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:31.266675 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 21:38:31.266857 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":338,"threads_started":1}
I20260927 21:38:31.270835 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:31.273815 26147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440: Bootstrap starting.
I20260927 21:38:31.276955 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:31.274719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43447 (local address 127.3.186.254:38275)
0927 21:38:31.274888 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:31.274897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:31.275325 (+ 428us) server_negotiation.cc:407] Connection header received
0927 21:38:31.275469 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:31.275472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:31.275509 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:31.275578 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:31.276387 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:31.276391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:31.276394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:38:31.276405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:31.276435 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:38:31.276447 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:31.276523 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:31.276659 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:38:31.276694 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:38:31.277199 26147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:31.278321 26147 log.cc:824] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:31.280604 26147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440: No bootstrap required, opened a new log
I20260927 21:38:31.287283 26147 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d0b9d94a384c4c83eb668a2710f440" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38275 } }
I20260927 21:38:31.287670 26147 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:38:31.287806 26147 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d0b9d94a384c4c83eb668a2710f440, State: Initialized, Role: FOLLOWER
I20260927 21:38:31.288379 26147 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d0b9d94a384c4c83eb668a2710f440" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38275 } }
I20260927 21:38:31.288625 26147 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:38:31.288766 26147 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:38:31.288931 26147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:31.290868 26147 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d0b9d94a384c4c83eb668a2710f440" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38275 } }
I20260927 21:38:31.291289 26147 leader_election.cc:302] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d0b9d94a384c4c83eb668a2710f440; no voters:
I20260927 21:38:31.291781 26147 leader_election.cc:288] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:38:31.291894 26157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:31.292658 26157 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [term 1 LEADER]: Becoming Leader. State: Replica: c6d0b9d94a384c4c83eb668a2710f440, State: Running, Role: LEADER
I20260927 21:38:31.293208 26157 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d0b9d94a384c4c83eb668a2710f440" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38275 } }
I20260927 21:38:31.293721 26147 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:31.296175 26158 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d0b9d94a384c4c83eb668a2710f440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d0b9d94a384c4c83eb668a2710f440" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38275 } } }
I20260927 21:38:31.296162 26159 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d0b9d94a384c4c83eb668a2710f440. Latest consensus state: current_term: 1 leader_uuid: "c6d0b9d94a384c4c83eb668a2710f440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d0b9d94a384c4c83eb668a2710f440" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 38275 } } }
I20260927 21:38:31.296648 26158 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:31.296687 26159 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:31.297803 26164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:31.303962 26164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:31.312186 26164 catalog_manager.cc:1347] Generated new cluster ID: 90a2fbf0b0394ab0ab0bf8153f59d06d
I20260927 21:38:31.312282 26164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:31.324860 26164 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:31.325238 26164 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:31.333940 26164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d0b9d94a384c4c83eb668a2710f440: Generated new TSK 0
I20260927 21:38:31.334708 26164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:31.344342 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26080
W20260927 21:38:31.345307 26142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41385
W20260927 21:38:31.345957 3819 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26080: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:38:31.346097 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26080
2026-09-27T21:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1727 ms)
[----------] 18 tests from AuthTokenIssuingTest (32350 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:31 dist-test-slave-pqr8 krb5kdc[26178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:31 dist-test-slave-pqr8 krb5kdc[26178](info): set up 2 sockets
Sep 27 21:38:31 dist-test-slave-pqr8 krb5kdc[26178](info): commencing operation
krb5kdc: starting...
W20260927 21:38:32.402064 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:32 dist-test-slave-pqr8 krb5kdc[26178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:32.613234 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43579
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:43579,127.3.186.253:35799,127.3.186.252:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:32.920449 26191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:38:32.920718 26191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:32.920778 26191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:32.920843 26191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:32.927609 26191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:32.927681 26191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:32.927742 26191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:32.927790 26191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:32.927843 26191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:32.936921 26191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:43579,127.3.186.253:35799,127.3.186.252:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:32.937983 26191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:32.939121 26191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:32.950083 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:32.950289 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:32.950532 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:32.951347 26191 server_base.cc:1034] running on GCE node
I20260927 21:38:32.952073 26191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:32.953265 26191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:32.954504 26191 hybrid_clock.cc:648] HybridClock initialized: now 1790545112954446 us; error 68 us; skew 500 ppm
I20260927 21:38:32.954841 26191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:32 dist-test-slave-pqr8 krb5kdc[26178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:32.963243 26191 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:32.964535 26191 webserver.cc:460] Webserver started at http://127.3.186.254:36027/ using document root <none> and password file <none>
I20260927 21:38:32.965150 26191 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:32.965273 26191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:32.965658 26191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:32.968473 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "2845804efb0c48408f676d7c3058d897"
format_stamp: "Formatted at 2026-09-27 21:38:32 on dist-test-slave-pqr8"
server_key: "02a9badac5c58c6771f599e62229126f"
server_key_iv: "1233b874447f9fbe727b3a432458c2ed"
server_key_version: "encryptionkey@0"
I20260927 21:38:32.969211 26191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "2845804efb0c48408f676d7c3058d897"
format_stamp: "Formatted at 2026-09-27 21:38:32 on dist-test-slave-pqr8"
server_key: "02a9badac5c58c6771f599e62229126f"
server_key_iv: "1233b874447f9fbe727b3a432458c2ed"
server_key_version: "encryptionkey@0"
I20260927 21:38:32.973829 26191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:38:32.976879 26206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:32.978121 26191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:32.978284 26191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "2845804efb0c48408f676d7c3058d897"
format_stamp: "Formatted at 2026-09-27 21:38:32 on dist-test-slave-pqr8"
server_key: "02a9badac5c58c6771f599e62229126f"
server_key_iv: "1233b874447f9fbe727b3a432458c2ed"
server_key_version: "encryptionkey@0"
I20260927 21:38:32.978485 26191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:32.995222 26191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:32.998418 26191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:32.998682 26191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:33.015727 26191 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:43579
I20260927 21:38:33.015754 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:43579 every 8 connection(s)
I20260927 21:38:33.016855 26191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:33.020011 26258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:33.020918 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26191
I20260927 21:38:33.021128 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:33.021899 3819 external_mini_cluster.cc:1428] Setting key 288390f0efefa64d5bdfb3cc08033845
I20260927 21:38:33.029217 26258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.186.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.253@KRBTEST.COM" created.
I20260927 21:38:33.050299 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.032511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43579 (local address 127.0.0.1:50418)
0927 21:38:33.033396 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.033440 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.035123 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.035863 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.035869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.036519 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.036817 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.036832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.038371 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.038380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.039395 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.039406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.040264 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.040271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.040497 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.041075 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.041099 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.043228 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.046331 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.046343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.046352 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.046748 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.047135 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.047141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.047144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.047304 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.047981 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.047988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.048216 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.048479 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.049092 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":182,"spinlock_wait_cycles":20352,"thread_start_us":414,"threads_started":1}
I20260927 21:38:33.050319 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.032510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.3.186.254:43579)
0927 21:38:33.033234 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.033246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.033572 (+ 326us) server_negotiation.cc:407] Connection header received
0927 21:38:33.035456 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.035461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.035653 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.035851 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.037016 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.038222 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.039532 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.040045 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.040236 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.043425 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.043432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.043441 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.043470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.046200 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.046873 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.046883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.046891 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.047032 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.047444 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.047450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.047453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.047851 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.048042 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.048606 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.048979 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":530,"thread_start_us":335,"threads_started":1}
I20260927 21:38:33.053017 26258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:33.053998 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.053634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33864)
0927 21:38:33.053791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.053901 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:38:33.054265 26207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
I20260927 21:38:33.055723 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.055371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33880)
0927 21:38:33.055525 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.055593 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:38:33.055925 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:35799: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab.
Entry for principal kudu/127.3.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab.
I20260927 21:38:33.076053 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:35799
--webserver_interface=127.3.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab
--principal=kudu/127.3.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:43579,127.3.186.253:35799,127.3.186.252:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:38:33.105178 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } attempt: 1
I20260927 21:38:33.106345 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.105864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33896)
0927 21:38:33.106065 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.106203 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 21:38:33.107334 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.106943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33906)
0927 21:38:33.107092 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.107184 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:38:33.107534 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:35799: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
I20260927 21:38:33.175772 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } attempt: 2
I20260927 21:38:33.176995 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.176445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33910)
0927 21:38:33.176724 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.176842 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 21:38:33.178054 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.177681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33918)
0927 21:38:33.177838 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.177914 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:38:33.178238 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:35799: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
I20260927 21:38:33.269491 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } attempt: 3
I20260927 21:38:33.270728 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.270195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33928)
0927 21:38:33.270420 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.270541 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:38:33.271910 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.271463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33940)
0927 21:38:33.271643 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.271740 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 21:38:33.272113 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:35799: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
W20260927 21:38:33.385186 26265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:38:33.385473 26265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:33.385654 26265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:33.385771 26265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:33.392576 26265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:33.392656 26265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:33.392764 26265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:33.392848 26265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:33.392930 26265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:33.402081 26265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:43579,127.3.186.253:35799,127.3.186.252:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab
--principal=kudu/127.3.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:35799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:33.403122 26265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:33.404345 26265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:33.415483 26270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:33.415555 26271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:33.415581 26273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:33.416064 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } attempt: 4
I20260927 21:38:33.416513 26265 server_base.cc:1034] running on GCE node
I20260927 21:38:33.417255 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.416682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33954)
0927 21:38:33.416965 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.417078 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260927 21:38:33.417289 26265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:33.418253 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.417863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33964)
0927 21:38:33.418017 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.418100 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:38:33.418437 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:35799: Network error: Client connection negotiation failed: client connection to 127.3.186.253:35799: connect: Connection refused (error 111)
I20260927 21:38:33.418844 26265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:33.420039 26265 hybrid_clock.cc:648] HybridClock initialized: now 1790545113419987 us; error 26 us; skew 500 ppm
I20260927 21:38:33.420296 26265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:33.423188 26265 init.cc:377] Logged in from keytab as kudu/127.3.186.253@KRBTEST.COM (short username kudu)
I20260927 21:38:33.424414 26265 webserver.cc:460] Webserver started at http://127.3.186.253:42105/ using document root <none> and password file <none>
I20260927 21:38:33.424971 26265 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:33.425050 26265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:33.425294 26265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:33.428004 26265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data/instance:
uuid: "fb7ed77c8b28416b923004eb03d67bdb"
format_stamp: "Formatted at 2026-09-27 21:38:33 on dist-test-slave-pqr8"
server_key: "41012fc835bf77b8226f4586491faffc"
server_key_iv: "9dc65f25ad27abe4448dfae2a5acea15"
server_key_version: "encryptionkey@0"
I20260927 21:38:33.428704 26265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal/instance:
uuid: "fb7ed77c8b28416b923004eb03d67bdb"
format_stamp: "Formatted at 2026-09-27 21:38:33 on dist-test-slave-pqr8"
server_key: "41012fc835bf77b8226f4586491faffc"
server_key_iv: "9dc65f25ad27abe4448dfae2a5acea15"
server_key_version: "encryptionkey@0"
I20260927 21:38:33.433231 26265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 21:38:33.436481 26280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:33.437788 26265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:38:33.437911 26265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal
uuid: "fb7ed77c8b28416b923004eb03d67bdb"
format_stamp: "Formatted at 2026-09-27 21:38:33 on dist-test-slave-pqr8"
server_key: "41012fc835bf77b8226f4586491faffc"
server_key_iv: "9dc65f25ad27abe4448dfae2a5acea15"
server_key_version: "encryptionkey@0"
I20260927 21:38:33.438060 26265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:33.456187 26265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:33.459834 26265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:33.460026 26265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:33.476873 26265 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.253:35799
I20260927 21:38:33.476903 26331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.253:35799 every 8 connection(s)
I20260927 21:38:33.478009 26265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
I20260927 21:38:33.481343 26332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:33.484598 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26265
I20260927 21:38:33.484781 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-1/wal/instance
I20260927 21:38:33.485271 3819 external_mini_cluster.cc:1428] Setting key 6b2b05e21f955d9208456fac633585d6
I20260927 21:38:33.489188 26332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:33.509379 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.492137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.3.186.254:43579)
0927 21:38:33.492356 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.492365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.493383 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 21:38:33.495123 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.495132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.495218 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.495333 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.496573 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.497236 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.499599 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.500005 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.500096 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.503843 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.503847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.503850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.503879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.506472 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.507254 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.507266 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.507268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.507376 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.507790 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.507794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.507797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.508034 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.508180 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.508854 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.509084 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:38:33.510212 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.492527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43579 (local address 127.0.0.1:50432)
0927 21:38:33.493222 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.493267 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.494875 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.495508 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.495515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.495960 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.496428 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.496442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.497394 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.497403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.499452 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.499469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.500220 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.500229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.500534 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.501153 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.501184 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.503598 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.506614 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.506623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.506635 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.507129 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.507483 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.507488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.507492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.507665 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.508153 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.508160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.508488 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.508743 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.509170 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":309,"threads_started":1}
WARNING: no policy specified for kudu/127.3.186.252@KRBTEST.COM; defaulting to no policy
I20260927 21:38:33.511725 26332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.252@KRBTEST.COM" created.
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:38:33.526198 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.512413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33980)
0927 21:38:33.512572 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.512586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.512659 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.513743 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.513748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.514147 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.514269 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.514277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.515545 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.515552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.516346 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.516356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.517090 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.517095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.517336 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.517917 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.517940 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.519930 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.523930 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.523933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.523935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.524239 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.524612 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.524615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.524617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.524762 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.525429 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.525433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.525643 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.525772 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.525939 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260927 21:38:33.526916 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.512642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.3.186.253:35799)
0927 21:38:33.513323 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.513328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.513351 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:38:33.513389 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.513393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.513598 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.513739 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.514434 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.515420 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.516476 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.516955 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.517090 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.520137 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.520151 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.520167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.520210 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.523801 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.524358 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.524367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.524373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.524510 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.524866 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.524874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.524880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.525298 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.525529 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.525965 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.526240 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":545,"thread_start_us":414,"threads_started":1}
I20260927 21:38:33.528894 26332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:33.529832 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.529387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33306)
0927 21:38:33.529557 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.529661 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:38:33.530035 26283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.531519 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.531084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33310)
0927 21:38:33.531266 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.531371 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 21:38:33.531762 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab.
Entry for principal kudu/127.3.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab.
I20260927 21:38:33.542424 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:44377
--webserver_interface=127.3.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab
--principal=kudu/127.3.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:43579,127.3.186.253:35799,127.3.186.252:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:38:33.581039 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 1
I20260927 21:38:33.582235 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.581749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33318)
0927 21:38:33.581955 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.582100 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 21:38:33.583148 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.582816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33322)
0927 21:38:33.582985 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.583053 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:38:33.583335 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.651532 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 2
I20260927 21:38:33.652676 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.652228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33336)
0927 21:38:33.652426 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.652509 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 21:38:33.653841 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.653417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33338)
0927 21:38:33.653658 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.653737 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 21:38:33.654063 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.717729 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } attempt: 5
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:38:33.730655 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.718547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.3.186.253:35799)
0927 21:38:33.718754 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.718762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.718870 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:38:33.719102 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.719106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.719168 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.719277 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.720057 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.720726 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.721926 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.722364 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.722499 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.725637 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.725642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.725646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.725676 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.728169 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.728791 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.728804 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.728807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.728918 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.729280 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.729285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.729289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.729614 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.729780 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.730133 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.730334 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 21:38:33.730656 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.718503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33994)
0927 21:38:33.718751 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.718773 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.718948 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.719317 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.719323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.719711 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.719909 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.719918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.720855 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.720862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.721783 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.721795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.722507 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.722512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.722701 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.723298 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.723315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.725422 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.728299 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.728305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.728309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.728671 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.729026 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.729031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.729035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.729171 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.729744 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.729748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.729893 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.730032 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.730321 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 21:38:33.731673 26258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:33.732833 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.732349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33354)
0927 21:38:33.732542 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.732647 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:38:33.733775 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.733442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33364)
0927 21:38:33.733584 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.733671 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 21:38:33.733961 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.745265 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 3
I20260927 21:38:33.746155 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.745782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33372)
0927 21:38:33.745959 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.746034 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:38:33.747133 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.746772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33378)
0927 21:38:33.746973 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.747034 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260927 21:38:33.747390 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.785192 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 1
I20260927 21:38:33.786355 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.785813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33388)
0927 21:38:33.786016 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.786161 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 21:38:33.787802 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.787420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33400)
0927 21:38:33.787592 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.787674 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 21:38:33.788116 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
W20260927 21:38:33.849579 26339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:38:33.849996 26339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:33.850078 26339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:33.850154 26339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:38:33.856361 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 2
W20260927 21:38:33.857039 26339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:33.857126 26339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:33.857213 26339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:33.857282 26339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:33.857366 26339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:33.857578 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.856984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33416)
0927 21:38:33.857238 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.857357 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:38:33.858600 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.858259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33432)
0927 21:38:33.858400 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.858488 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:38:33.858803 26258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.866902 26339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:43579,127.3.186.253:35799,127.3.186.252:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab
--principal=kudu/127.3.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:33.867900 26339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:33.869057 26339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:33.879797 26344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:33.879832 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:33.879926 26347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:33.881040 26339 server_base.cc:1034] running on GCE node
I20260927 21:38:33.881774 26339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:33.882987 26339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:33.884188 26339 hybrid_clock.cc:648] HybridClock initialized: now 1790545113884134 us; error 25 us; skew 500 ppm
I20260927 21:38:33.884485 26339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:33.887559 26339 init.cc:377] Logged in from keytab as kudu/127.3.186.252@KRBTEST.COM (short username kudu)
I20260927 21:38:33.888847 26339 webserver.cc:460] Webserver started at http://127.3.186.252:32777/ using document root <none> and password file <none>
I20260927 21:38:33.889400 26339 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:33.889478 26339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:33.889784 26339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:33.890579 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 4
I20260927 21:38:33.891598 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.891174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33442)
0927 21:38:33.891360 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.891434 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 21:38:33.892496 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.892201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33444)
0927 21:38:33.892335 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.892404 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:38:33.892735 26332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:44377: Network error: Client connection negotiation failed: client connection to 127.3.186.252:44377: connect: Connection refused (error 111)
I20260927 21:38:33.892743 26339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data/instance:
uuid: "b31a332f791b408fa9ceda6dd3a7e90f"
format_stamp: "Formatted at 2026-09-27 21:38:33 on dist-test-slave-pqr8"
server_key: "ccc19b2c1025f57d0bc2c2c40a7266d3"
server_key_iv: "2b578f2b3baa7d2c2488bd7e95fd84ff"
server_key_version: "encryptionkey@0"
I20260927 21:38:33.893484 26339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal/instance:
uuid: "b31a332f791b408fa9ceda6dd3a7e90f"
format_stamp: "Formatted at 2026-09-27 21:38:33 on dist-test-slave-pqr8"
server_key: "ccc19b2c1025f57d0bc2c2c40a7266d3"
server_key_iv: "2b578f2b3baa7d2c2488bd7e95fd84ff"
server_key_version: "encryptionkey@0"
I20260927 21:38:33.897962 26339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 21:38:33.901080 26354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:33.902220 26339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:33.902379 26339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal
uuid: "b31a332f791b408fa9ceda6dd3a7e90f"
format_stamp: "Formatted at 2026-09-27 21:38:33 on dist-test-slave-pqr8"
server_key: "ccc19b2c1025f57d0bc2c2c40a7266d3"
server_key_iv: "2b578f2b3baa7d2c2488bd7e95fd84ff"
server_key_version: "encryptionkey@0"
I20260927 21:38:33.902593 26339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:33.920251 26339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:33.924137 26339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:33.924371 26339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:33.941928 26339 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.252:44377
I20260927 21:38:33.941998 26405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.252:44377 every 8 connection(s)
I20260927 21:38:33.943008 26339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
I20260927 21:38:33.946305 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:33.951398 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26339
I20260927 21:38:33.951563 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544773327888-3819-0/minicluster-data/master-2/wal/instance
I20260927 21:38:33.951922 3819 external_mini_cluster.cc:1428] Setting key e6ebb1063a0fdf5721e8e8ee20584cf9
I20260927 21:38:33.952247 26406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:33.965005 26258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 3
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:38:33.972770 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.954947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.3.186.254:43579)
0927 21:38:33.955172 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.955180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.955935 (+ 755us) server_negotiation.cc:407] Connection header received
0927 21:38:33.956439 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.956448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.956522 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.956641 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.958245 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.959403 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.961321 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.961885 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.962012 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.965533 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.965541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.965546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.965603 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.969350 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.970206 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.970213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.970217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.970397 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.970807 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.970812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.970816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.971070 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.971221 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.971909 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.972384 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:38:33.973555 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.953581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.3.186.252:44377)
0927 21:38:33.954311 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.954323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.954362 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:38:33.956210 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.956228 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.956481 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.956668 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.958343 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.959950 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.961579 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.962294 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.962382 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.966942 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.966954 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.966967 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.967003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.970429 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.971238 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.971249 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.971254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.971403 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.971825 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.971830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.971833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.972270 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.972530 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.973057 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.973202 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260927 21:38:33.973601 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.955013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:43579 (local address 127.0.0.1:50444)
0927 21:38:33.955783 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.955828 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.956220 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.956696 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.956702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.957556 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.958044 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.958066 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.959560 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.959564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.961182 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.961197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.962070 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.962075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.962225 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.962838 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.962870 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.965276 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.969471 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.969483 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.969493 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.970043 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.970507 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.970513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.970516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.970698 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.971198 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.971205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.971469 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.971724 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.972150 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":1,"thread_start_us":378,"threads_started":1}
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:38:33.975596 26406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:33.981791 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.965731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.3.186.252:44377)
0927 21:38:33.966338 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.966341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.966357 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:38:33.966390 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.966392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.966432 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.966544 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.967772 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.969056 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.970880 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.971540 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.971661 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.975247 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.975254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.975258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.975290 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.979074 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.979777 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.979780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.979782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.979885 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.980310 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.980314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.980317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.980593 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.980741 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.981181 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.981370 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":388,"threads_started":1}
I20260927 21:38:33.982182 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.965593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33458)
0927 21:38:33.965783 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.965801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.965902 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.966856 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.966860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.967422 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.967638 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.967647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.969205 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.969211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.970720 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.970733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.971683 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.971691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.971863 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.972403 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.972430 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.975036 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.979246 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.979251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.979255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.979642 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.979995 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.980003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.980007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.980188 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.980727 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.980732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.980901 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.981033 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.981265 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:33.987875 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897: Bootstrap starting.
I20260927 21:38:33.990078 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.976307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:35799 (local address 127.0.0.1:33996)
0927 21:38:33.976510 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.976525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.976614 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.977137 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.977142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.977546 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.977788 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.977803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.978785 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.978792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.980006 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.980020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.980912 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.980918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.981195 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.981767 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.981794 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.984344 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:33.987507 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.987514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.987518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.988005 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.988439 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:33.988444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:33.988447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:33.988624 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:33.989241 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:33.989246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:33.989381 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:38:33.989518 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 21:38:33.989774 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 21:38:33.990324 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.976365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.3.186.253:35799)
0927 21:38:33.976518 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.976525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.976569 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:38:33.976769 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.976774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.976845 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.976944 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.977981 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.978648 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.980155 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.980605 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.980735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.984509 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.984513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.984516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.984559 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.987355 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.988125 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.988132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.988136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.988298 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.988763 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.988769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.988774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.989113 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.989275 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.989638 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.989850 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:38:33.991158 26406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:33.991935 26258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:33.993232 26258 log.cc:824] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:33.994925 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.977142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.3.186.254:43579)
0927 21:38:33.977290 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.977297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.977316 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:38:33.977732 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.977738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.977815 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.977900 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.979430 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.980603 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.982200 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.982831 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.982950 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.988417 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.988422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.988425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.988450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:33.992169 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.992953 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.992961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.992967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.993122 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:33.993567 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:33.993571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:33.993574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:33.993993 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:33.994286 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:33.994550 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:38:33.994686 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:38:33.996641 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897: No bootstrap required, opened a new log
Sep 27 21:38:33 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
Sep 27 21:38:34 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:38:34.005301 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.991724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33460)
0927 21:38:33.991955 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:33.991966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:33.992133 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:33.992414 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:33.992418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:33.992746 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:33.992919 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.992932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.994162 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.994166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.995413 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:33.995427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.996347 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.996355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:33.996515 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.997082 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:33.997100 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:33.999457 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:34.002889 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:34.002897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:34.002900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:34.003391 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:34.003717 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:34.003723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:34.003726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:34.003846 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:34.004340 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:34.004344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:34.004472 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 21:38:34.004611 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:38:34.004915 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260927 21:38:34.005391 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.991738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.3.186.252:44377)
0927 21:38:33.991900 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.991903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.992032 (+ 129us) server_negotiation.cc:407] Connection header received
0927 21:38:33.992239 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.992243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.992312 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.992398 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.993059 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.994021 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.995579 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.996207 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:33.996928 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:33.999695 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:33.999702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:33.999706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:33.999733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:34.002751 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.003485 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.003493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.003496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.003610 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.003940 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.003947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.003951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.004223 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:34.004357 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:34.004687 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:38:34.005046 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":89}
I20260927 21:38:34.007189 26258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.007513 26258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2845804efb0c48408f676d7c3058d897, State: Initialized, Role: FOLLOWER
I20260927 21:38:34.007953 26258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.008917 26413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.009411 26413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:34.010028 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: Bootstrap starting.
I20260927 21:38:34.010145 26258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:34.012506 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:33.996600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.3.186.253:35799)
0927 21:38:33.996759 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:33.996765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:33.996781 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:38:33.996820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:33.996824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:33.996919 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:33.997007 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:33.998257 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:33.999013 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.000136 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.000666 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.000794 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:34.005081 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:34.005087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:34.005091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:34.005125 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:34.008589 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.009344 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.009349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.009352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.009502 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.009997 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.010002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.010005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.010344 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:34.010601 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:34.011746 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0927 21:38:34.012027 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:38:34.013700 26406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:34.014761 26406 log.cc:824] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:34.017788 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:34.021905 26233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2845804efb0c48408f676d7c3058d897"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50458
I20260927 21:38:34.022202 26233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:38:34.022383 26233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:34.023464 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: No bootstrap required, opened a new log
I20260927 21:38:34.024632 26233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.027803 26233 leader_election.cc:288] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [CANDIDATE]: Term 1 election: Requested vote from peers fb7ed77c8b28416b923004eb03d67bdb (127.3.186.253:35799), b31a332f791b408fa9ceda6dd3a7e90f (127.3.186.252:44377)
I20260927 21:38:34.033690 26406 raft_consensus.cc:348] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.034019 26406 raft_consensus.cc:729] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31a332f791b408fa9ceda6dd3a7e90f, State: Initialized, Role: FOLLOWER
I20260927 21:38:34.034468 26406 consensus_queue.cc:260] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.035630 26424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.036409 26424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:34.037007 26406 sys_catalog.cc:564] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:34.037042 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2845804efb0c48408f676d7c3058d897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b31a332f791b408fa9ceda6dd3a7e90f"
I20260927 21:38:34.037525 26381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:34.042675 26381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2845804efb0c48408f676d7c3058d897 in term 1.
I20260927 21:38:34.043659 26209 leader_election.cc:302] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2845804efb0c48408f676d7c3058d897, b31a332f791b408fa9ceda6dd3a7e90f; no voters:
I20260927 21:38:34.044174 26413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:34.056497 26413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [term 1 LEADER]: Becoming Leader. State: Replica: 2845804efb0c48408f676d7c3058d897, State: Running, Role: LEADER
I20260927 21:38:34.057560 26413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.074875 26430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2845804efb0c48408f676d7c3058d897. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.075348 26430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:34.082257 26443 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:38:34.086160 26444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:34.091857 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:34.056002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.3.186.253:35799)
0927 21:38:34.056219 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.056228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.057637 (+ 1409us) server_negotiation.cc:407] Connection header received
0927 21:38:34.061199 (+ 3562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.061208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.061289 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.061372 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.070371 (+ 8999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.071785 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.075709 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.076576 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.076707 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:34.084733 (+ 8026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:34.084742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:34.084750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:34.084801 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:34.088552 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.089447 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.089453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.089456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.089622 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.090072 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.090079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.090084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.090435 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:34.090649 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:34.091220 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 21:38:34.091468 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:38:34.095734 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:34.057773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.3.186.252:44377)
0927 21:38:34.058010 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.058015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.066698 (+ 8683us) server_negotiation.cc:407] Connection header received
0927 21:38:34.067085 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.067092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.067187 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.067267 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.069599 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.070497 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.072669 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.073323 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.073452 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:34.082574 (+ 9122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:34.082582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:34.082585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:34.082621 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:34.086137 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.092886 (+ 6749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.092891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.092893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.093045 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.094451 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.094455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.094458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.094701 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:34.095020 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:34.095236 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:38:34.095428 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:38:34.098635 26444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:34.107244 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:34.052571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.3.186.254:43579)
0927 21:38:34.052786 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.052795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.056911 (+ 4116us) server_negotiation.cc:407] Connection header received
0927 21:38:34.058413 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.058419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.058498 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.058571 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.066051 (+ 7480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.067460 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.070071 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.070688 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.070796 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:34.094506 (+ 23710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:34.094514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:34.094522 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:34.094579 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:34.100965 (+ 6386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.102518 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.102528 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.102533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.102710 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.104272 (+ 1562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.104286 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.104290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.104626 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:34.104876 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:34.106165 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0927 21:38:34.106563 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:38:34.086329 26191 master_runner.cc:186] Error getting master registration for 127.3.186.253:35799: OK, instance_id { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" instance_seqno: 1790545113471938 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:38:34.114605 26191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:38:34.122570 26381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [term 1 FOLLOWER]: Refusing update from remote peer 2845804efb0c48408f676d7c3058d897: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:38:34.123523 26430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:38:34.116446 26339 master_runner.cc:186] Error getting master registration for 127.3.186.253:35799: OK, instance_id { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" instance_seqno: 1790545113471938 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:38:34.126855 26339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:38:34.131644 26448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:38:34.131768 26448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:38:34.147908 26424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2845804efb0c48408f676d7c3058d897. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.148195 26424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:34.150774 26430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.151021 26430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:34.153903 26444 catalog_manager.cc:1347] Generated new cluster ID: a9f84659df274d7db2f243d9e1fd4ef9
I20260927 21:38:34.153999 26444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:34.154505 26424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.154718 26424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:34.174401 26444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:38:34.176009 26444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:34.190357 26444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897: Generated new TSK 0
I20260927 21:38:34.191784 26444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:34.191934 26332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } attempt: 5
Sep 27 21:38:34 dist-test-slave-pqr8 krb5kdc[26178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545113, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:38:34.214020 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:34.192739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.3.186.252:44377)
0927 21:38:34.192949 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.192955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.192972 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:38:34.193020 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.193024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.193089 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.193190 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.194183 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.195432 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.197088 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.197890 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.198019 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:34.202151 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:34.202159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:34.202164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:34.202199 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:34.208415 (+ 6216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.209338 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.209344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.209347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.209590 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:34.210409 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:34.210414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:34.210418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:34.211093 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:34.211299 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:34.212185 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 21:38:34.212611 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 21:38:34.214936 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:34.192437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:44377 (local address 127.0.0.1:33484)
0927 21:38:34.192602 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:34.192630 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:34.192732 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:34.193228 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:34.193234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:34.193764 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:34.193968 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:34.193984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.195557 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.195563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:34.196945 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:34.196959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:34.198215 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:34.198221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:34.198435 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:34.199100 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:34.199133 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:34.201949 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:34.208632 (+ 6683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:34.208638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:34.208643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:34.209158 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:34.209904 (+ 746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:34.209909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:34.209912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:34.210175 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:34.211466 (+ 1291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:34.211472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:34.211736 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 21:38:34.212682 (+ 946us) client_negotiation.cc:240] Negotiation successful
0927 21:38:34.213823 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 21:38:34.220311 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: Bootstrap starting.
I20260927 21:38:34.224014 26332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:34.225188 26332 log.cc:824] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:34.238478 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: No bootstrap required, opened a new log
I20260927 21:38:34.256911 26332 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.257649 26332 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7ed77c8b28416b923004eb03d67bdb, State: Initialized, Role: FOLLOWER
I20260927 21:38:34.258510 26332 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } }
I20260927 21:38:34.264617 26307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2845804efb0c48408f676d7c3058d897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb7ed77c8b28416b923004eb03d67bdb"
I20260927 21:38:34.265107 26306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:34.268508 26306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [term 1 FOLLOWER]: Refusing update from remote peer 2845804efb0c48408f676d7c3058d897: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:38:34.269449 26430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:38:34.273301 26307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2845804efb0c48408f676d7c3058d897 in term 1.
I20260927 21:38:34.279512 26332 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:38:34.326910 26260 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.307788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.3.186.254:43579)
0927 21:38:34.307960 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.307966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.324993 (+ 17027us) server_negotiation.cc:407] Connection header received
0927 21:38:34.325433 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.325440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.325572 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.325657 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.326670 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50476: BlockingRecv error: recv got EOF from 127.0.0.1:50476 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 21:38:34.328997 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.309602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.3.186.253:35799)
0927 21:38:34.309768 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.309774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.327546 (+ 17772us) server_negotiation.cc:407] Connection header received
0927 21:38:34.327800 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.327805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.327874 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.327941 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.328806 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34022: BlockingRecv error: recv got EOF from 127.0.0.1:34022 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 21:38:34.332656 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.330287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.3.186.252:44377)
0927 21:38:34.330436 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.330442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.330923 (+ 481us) server_negotiation.cc:407] Connection header received
0927 21:38:34.331121 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.331126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.331194 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.331319 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.332364 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33486: BlockingRecv error: recv got EOF from 127.0.0.1:33486 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:38:34.333354 26458 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.333756 26458 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:34.334123 26460 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2845804efb0c48408f676d7c3058d897. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.334384 26460 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:34.334645 26469 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.334946 26469 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:38:34.370036 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.357571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.3.186.253:35799)
0927 21:38:34.357717 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.357723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.368274 (+ 10551us) server_negotiation.cc:407] Connection header received
0927 21:38:34.368683 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.368689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.368797 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.368864 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.369762 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34036: BlockingRecv error: recv got EOF from 127.0.0.1:34036 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:38:34.389216 26487 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 21:38:34.400944 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.399707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.3.186.253:35799)
0927 21:38:34.399857 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.399863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.399879 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:38:34.399931 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.399935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.400008 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.400102 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.400769 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34048: BlockingRecv error: recv got EOF from 127.0.0.1:34048 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 21:38:34.408536 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.406690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.3.186.253:35799)
0927 21:38:34.406836 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.406841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.407268 (+ 427us) server_negotiation.cc:407] Connection header received
0927 21:38:34.407408 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.407413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.407494 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.407639 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.408330 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34054: BlockingRecv error: recv got EOF from 127.0.0.1:34054 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 21:38:34.417212 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.415690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.3.186.253:35799)
0927 21:38:34.415860 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.415867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.415991 (+ 124us) server_negotiation.cc:407] Connection header received
0927 21:38:34.416208 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.416213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.416289 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.416374 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.417017 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34056: BlockingRecv error: recv got EOF from 127.0.0.1:34056 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 21:38:34.430544 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.428334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.3.186.253:35799)
0927 21:38:34.428481 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.428486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.428947 (+ 461us) server_negotiation.cc:407] Connection header received
0927 21:38:34.429094 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.429099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.429173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.429269 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.430299 (+ 1030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57782: BlockingRecv error: recv got EOF from 127.0.0.1:57782 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 21:38:34.453178 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.451199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.3.186.253:35799)
0927 21:38:34.451362 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.451369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.451904 (+ 535us) server_negotiation.cc:407] Connection header received
0927 21:38:34.452055 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.452059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.452135 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.452225 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.452981 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57796: BlockingRecv error: recv got EOF from 127.0.0.1:57796 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 21:38:34.508014 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.505727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.3.186.253:35799)
0927 21:38:34.505900 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.505906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.506445 (+ 539us) server_negotiation.cc:407] Connection header received
0927 21:38:34.506718 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.506724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.506817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.506932 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.507782 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57798: BlockingRecv error: recv got EOF from 127.0.0.1:57798 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 21:38:34.577673 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.575419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.3.186.253:35799)
0927 21:38:34.575598 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.575605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.576146 (+ 541us) server_negotiation.cc:407] Connection header received
0927 21:38:34.576349 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.576356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.576450 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.576559 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.577416 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:38:34.658030 26430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2845804efb0c48408f676d7c3058d897" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2845804efb0c48408f676d7c3058d897" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 43579 } } peers { permanent_uuid: "fb7ed77c8b28416b923004eb03d67bdb" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 35799 } } peers { permanent_uuid: "b31a332f791b408fa9ceda6dd3a7e90f" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 44377 } } }
I20260927 21:38:34.658883 26430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2845804efb0c48408f676d7c3058d897 [sys.catalog]: This master's current role is: LEADER
W20260927 21:38:34.710767 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.708830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.3.186.253:35799)
0927 21:38:34.709024 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.709029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.709401 (+ 372us) server_negotiation.cc:407] Connection header received
0927 21:38:34.709599 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.709603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.709710 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.709797 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.710465 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57812: BlockingRecv error: recv got EOF from 127.0.0.1:57812 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 21:38:34.971662 26336 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:34.970136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.3.186.253:35799)
0927 21:38:34.970357 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:34.970364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:34.970514 (+ 150us) server_negotiation.cc:407] Connection header received
0927 21:38:34.970682 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:34.970686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:34.970755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:34.970839 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:34.971469 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57814: BlockingRecv error: recv got EOF from 127.0.0.1:57814 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:38:35.133893 26448 catalog_manager.cc:1259] Loaded cluster ID: a9f84659df274d7db2f243d9e1fd4ef9
I20260927 21:38:35.133980 26448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: loading cluster ID for follower catalog manager: success
I20260927 21:38:35.136322 26448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: acquiring CA information for follower catalog manager: success
I20260927 21:38:35.138121 26448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b31a332f791b408fa9ceda6dd3a7e90f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:38:35.393157 26487 catalog_manager.cc:1259] Loaded cluster ID: a9f84659df274d7db2f243d9e1fd4ef9
I20260927 21:38:35.393263 26487 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: loading cluster ID for follower catalog manager: success
I20260927 21:38:35.395701 26487 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: acquiring CA information for follower catalog manager: success
I20260927 21:38:35.397410 26487 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb7ed77c8b28416b923004eb03d67bdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:38:35.502594 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:35.492525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.3.186.253:35799)
0927 21:38:35.494284 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:35.494290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:35.494308 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:38:35.494375 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:35.494380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:35.494610 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:35.494881 (+ 271us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:35.495705 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.497230 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.499300 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.500121 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.500268 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:35.500902 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:35.501676 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:35.501831 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:35.501964 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:38:35.502052 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"thread_start_us":1462,"threads_started":1}
I20260927 21:38:35.526952 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:35.510328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.3.186.252:44377)
0927 21:38:35.519507 (+ 9179us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:35.519514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:35.519909 (+ 395us) server_negotiation.cc:407] Connection header received
0927 21:38:35.520136 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:35.520142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:35.520321 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:35.520483 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:35.520986 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.522024 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.524169 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.524783 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.524904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:35.525484 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:35.526160 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:35.526304 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:35.526403 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:38:35.526478 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9044,"thread_start_us":8955,"threads_started":1}
I20260927 21:38:35.559454 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:35.542755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.3.186.253:35799)
0927 21:38:35.542947 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:35.542954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:35.553082 (+ 10128us) server_negotiation.cc:407] Connection header received
0927 21:38:35.553326 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:35.553331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:35.553419 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:35.553485 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:35.554257 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.555397 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.557392 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.558016 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.558102 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:35.558741 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:35.558831 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:35.558940 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:35.559080 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:38:35.559165 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:38:35.560719 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:35.553264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.3.186.252:44377)
0927 21:38:35.553454 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:35.553461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:35.554363 (+ 902us) server_negotiation.cc:407] Connection header received
0927 21:38:35.554588 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:35.554593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:35.554682 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:35.554744 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:35.555388 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.556414 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.558663 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.559302 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.559433 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:35.559991 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:35.560121 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:35.560182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:35.560398 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:38:35.560473 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:38:35.563813 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:35.542419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.3.186.254:43579)
0927 21:38:35.553852 (+ 11433us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:35.553858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:35.553875 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:38:35.553984 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:35.553991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:35.554199 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:35.554386 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:35.554960 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.556023 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.561674 (+ 5651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:35.562315 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:35.562437 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:35.562845 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:35.563235 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:35.563340 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:35.563447 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:38:35.563526 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11309,"thread_start_us":11210,"threads_started":1}
I20260927 21:38:35.568534 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26191
I20260927 21:38:35.569680 26253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:35.676388 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26191
W20260927 21:38:35.696705 26284 connection.cc:535] server connection from 127.0.0.1:33994 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:38:35.697083 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26265
I20260927 21:38:35.698208 26327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:35.811479 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26265
I20260927 21:38:35.829934 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26339
I20260927 21:38:35.831023 26401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:35.931633 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26339
2026-09-27T21:38:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4591 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:38:36 dist-test-slave-pqr8 krb5kdc[26559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:38:36 dist-test-slave-pqr8 krb5kdc[26559](info): set up 2 sockets
Sep 27 21:38:36 dist-test-slave-pqr8 krb5kdc[26559](info): commencing operation
krb5kdc: starting...
W20260927 21:38:36.998031 3819 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/test-user.keytab.
Sep 27 21:38:37 dist-test-slave-pqr8 krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:38:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
Entry for principal kudu/127.3.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab.
I20260927 21:38:37.208912 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40993
--webserver_interface=127.3.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:40993,127.3.186.253:43709,127.3.186.252:45329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:38:37.512286 26572 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:38:37.512571 26572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:37.512689 26572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:37.512811 26572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:37.519483 26572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:37.519555 26572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:37.519651 26572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:37.519737 26572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:37.519860 26572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:37.528771 26572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:40993,127.3.186.253:43709,127.3.186.252:45329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.254.keytab
--principal=kudu/127.3.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.254:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:37.529834 26572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:37.531030 26572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:37.541950 26577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:37.541990 26580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:37.542026 26578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:37.542834 26572 server_base.cc:1034] running on GCE node
I20260927 21:38:37.543560 26572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:37.544664 26572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:37.545890 26572 hybrid_clock.cc:648] HybridClock initialized: now 1790545117545831 us; error 52 us; skew 500 ppm
I20260927 21:38:37.546188 26572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:37 dist-test-slave-pqr8 krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:37.548903 26572 init.cc:377] Logged in from keytab as kudu/127.3.186.254@KRBTEST.COM (short username kudu)
I20260927 21:38:37.550262 26572 webserver.cc:460] Webserver started at http://127.3.186.254:39421/ using document root <none> and password file <none>
I20260927 21:38:37.550827 26572 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:37.550961 26572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:37.551254 26572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:37.553869 26572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data/instance:
uuid: "102f6f5c101a48ba8fd5898502134bf9"
format_stamp: "Formatted at 2026-09-27 21:38:37 on dist-test-slave-pqr8"
server_key: "62482132bbfa40983620e185967ed0f1"
server_key_iv: "3c11a78aee23d91c48985a22c1480bcf"
server_key_version: "encryptionkey@0"
I20260927 21:38:37.554596 26572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal/instance:
uuid: "102f6f5c101a48ba8fd5898502134bf9"
format_stamp: "Formatted at 2026-09-27 21:38:37 on dist-test-slave-pqr8"
server_key: "62482132bbfa40983620e185967ed0f1"
server_key_iv: "3c11a78aee23d91c48985a22c1480bcf"
server_key_version: "encryptionkey@0"
I20260927 21:38:37.559074 26572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:38:37.562213 26587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:37.563342 26572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:38:37.563509 26572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal
uuid: "102f6f5c101a48ba8fd5898502134bf9"
format_stamp: "Formatted at 2026-09-27 21:38:37 on dist-test-slave-pqr8"
server_key: "62482132bbfa40983620e185967ed0f1"
server_key_iv: "3c11a78aee23d91c48985a22c1480bcf"
server_key_version: "encryptionkey@0"
I20260927 21:38:37.563719 26572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:37.575500 26572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:37.578860 26572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:37.579110 26572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:37.596625 26572 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.254:40993
I20260927 21:38:37.596632 26638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.254:40993 every 8 connection(s)
I20260927 21:38:37.597816 26572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/data/info.pb
I20260927 21:38:37.600948 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:37.605922 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26572
I20260927 21:38:37.606097 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-0/wal/instance
I20260927 21:38:37.606622 3819 external_mini_cluster.cc:1428] Setting key 48620b1891d06ab21c0acbafbc54fadb
I20260927 21:38:37.607169 26639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:37 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:37.630153 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.610676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40993 (local address 127.0.0.1:55780)
0927 21:38:37.611552 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.611620 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:37.614032 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:37.614770 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:37.614778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:37.615442 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:37.615874 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:37.615898 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:37.617368 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:37.617371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:37.618448 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:37.618459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:37.619738 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:37.619743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:37.619864 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:37.620449 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:37.620468 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:37.622511 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:37.626665 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:37.626674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:37.626681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:37.627028 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:37.627460 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:37.627463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:37.627465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:37.627590 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:37.628459 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:37.628464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:37.628666 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 21:38:37.628850 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 21:38:37.629154 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":421,"threads_started":1}
I20260927 21:38:37.630235 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.610676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.3.186.254:40993)
0927 21:38:37.612387 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:37.612393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:37.612436 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:38:37.614338 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:37.614343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:37.614537 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:37.614694 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:37.616065 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:37.617235 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:37.618587 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:37.619509 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:37.619706 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:37.622721 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:37.622734 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:37.622751 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:37.622791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:37.626527 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:37.627167 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:37.627177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:37.627183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:37.627358 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:37.627725 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:37.627732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:37.627738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:37.628325 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:37.628508 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:37.629070 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 21:38:37.629345 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552,"spinlock_wait_cycles":20736,"thread_start_us":1347,"threads_started":1}
WARNING: no policy specified for kudu/127.3.186.253@KRBTEST.COM; defaulting to no policy
I20260927 21:38:37.632728 26639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:37.633730 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.633331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53334)
0927 21:38:37.633498 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.633638 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:38:37.633972 26591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.253@KRBTEST.COM" created.
I20260927 21:38:37.635380 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.634987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53344)
0927 21:38:37.635136 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.635223 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 21:38:37.635588 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:43709: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab.
Entry for principal kudu/127.3.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab.
I20260927 21:38:37.660153 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:43709
--webserver_interface=127.3.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab
--principal=kudu/127.3.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:40993,127.3.186.253:43709,127.3.186.252:45329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:38:37.684896 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } attempt: 1
I20260927 21:38:37.685936 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.685478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53356)
0927 21:38:37.685719 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.685810 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260927 21:38:37.686935 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.686588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53370)
0927 21:38:37.686756 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.686825 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 21:38:37.687117 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:43709: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
I20260927 21:38:37.755332 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } attempt: 2
I20260927 21:38:37.756426 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.756007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53374)
0927 21:38:37.756219 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.756300 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 21:38:37.757395 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.757070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53376)
0927 21:38:37.757216 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.757285 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:38:37.757638 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:43709: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
I20260927 21:38:37.848887 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } attempt: 3
I20260927 21:38:37.850153 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.849685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53388)
0927 21:38:37.849929 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.850018 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 21:38:37.851229 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.850858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53398)
0927 21:38:37.851023 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.851098 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:38:37.851424 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:43709: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
W20260927 21:38:37.961891 26646 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:38:37.962219 26646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:37.962333 26646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:37.962458 26646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:37.969158 26646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:37.969259 26646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:37.969370 26646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:37.969449 26646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:37.969568 26646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:37.978463 26646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:40993,127.3.186.253:43709,127.3.186.252:45329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.253.keytab
--principal=kudu/127.3.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.253:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:37.979497 26646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:37.980696 26646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:38:37.991441 26651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:37.991519 26652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:37.991496 26654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:37.992141 26646 server_base.cc:1034] running on GCE node
I20260927 21:38:37.992806 26646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:37.994069 26646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:37.994665 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } attempt: 4
I20260927 21:38:37.995275 26646 hybrid_clock.cc:648] HybridClock initialized: now 1790545117995221 us; error 25 us; skew 500 ppm
I20260927 21:38:37.995568 26646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:38:37.995643 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.995175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53404)
0927 21:38:37.995437 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.995514 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 21:38:37.996675 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:37.996338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53408)
0927 21:38:37.996491 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:37.996566 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:38:37.996896 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.253:43709: Network error: Client connection negotiation failed: client connection to 127.3.186.253:43709: connect: Connection refused (error 111)
Sep 27 21:38:37 dist-test-slave-pqr8 krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:38.001754 26646 init.cc:377] Logged in from keytab as kudu/127.3.186.253@KRBTEST.COM (short username kudu)
I20260927 21:38:38.003014 26646 webserver.cc:460] Webserver started at http://127.3.186.253:36785/ using document root <none> and password file <none>
I20260927 21:38:38.003603 26646 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:38.003728 26646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:38.004012 26646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:38.006654 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data/instance:
uuid: "776a9358b3c1494190439d2b07d1482b"
format_stamp: "Formatted at 2026-09-27 21:38:38 on dist-test-slave-pqr8"
server_key: "ef7c36f80f07da93dd43ed2d061fb5ab"
server_key_iv: "599f16ea21f4229ea70a97ee28d8da4a"
server_key_version: "encryptionkey@0"
I20260927 21:38:38.007365 26646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal/instance:
uuid: "776a9358b3c1494190439d2b07d1482b"
format_stamp: "Formatted at 2026-09-27 21:38:38 on dist-test-slave-pqr8"
server_key: "ef7c36f80f07da93dd43ed2d061fb5ab"
server_key_iv: "599f16ea21f4229ea70a97ee28d8da4a"
server_key_version: "encryptionkey@0"
I20260927 21:38:38.011981 26646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:38:38.015098 26661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:38.016248 26646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:38:38.016402 26646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal
uuid: "776a9358b3c1494190439d2b07d1482b"
format_stamp: "Formatted at 2026-09-27 21:38:38 on dist-test-slave-pqr8"
server_key: "ef7c36f80f07da93dd43ed2d061fb5ab"
server_key_iv: "599f16ea21f4229ea70a97ee28d8da4a"
server_key_version: "encryptionkey@0"
I20260927 21:38:38.016587 26646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:38.030587 26646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:38.033906 26646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:38.034157 26646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:38.051643 26646 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.253:43709
I20260927 21:38:38.051713 26712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.253:43709 every 8 connection(s)
I20260927 21:38:38.052776 26646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/data/info.pb
I20260927 21:38:38.055998 26713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:38.058383 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26646
I20260927 21:38:38.058566 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-1/wal/instance
I20260927 21:38:38.058990 3819 external_mini_cluster.cc:1428] Setting key c5561cd2252df0b9f769c7072c359f81
I20260927 21:38:38.062011 26713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:38.081869 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.064194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.3.186.254:40993)
0927 21:38:38.064412 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.064420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.065414 (+ 994us) server_negotiation.cc:407] Connection header received
0927 21:38:38.067520 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.067529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.067653 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.067758 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.069138 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.070297 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.071644 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.072353 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.072485 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.076033 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.076037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.076040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.076088 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.078865 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.079791 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.079795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.079797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.079918 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.080347 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.080350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.080352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.080591 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.080696 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.081464 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.081668 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.3.186.252@KRBTEST.COM; defaulting to no policy
I20260927 21:38:38.083009 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.064527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40993 (local address 127.0.0.1:55786)
0927 21:38:38.065244 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.065296 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.067263 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.067983 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.067990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.068475 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.068979 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.068994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.070433 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.070437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.071503 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.071514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.072540 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.072544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.072714 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.073320 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.073344 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.075848 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.079001 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.079012 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.079030 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.079683 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.080019 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.080024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.080028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.080245 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.080718 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.080727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.081101 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.081402 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.081882 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.186.252@KRBTEST.COM" created.
I20260927 21:38:38.084555 26713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:38:38.097038 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.085282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53414)
0927 21:38:38.085417 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.085435 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.085565 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.086237 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.086242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.086645 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.086754 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.086762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.087970 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.087977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.088772 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.088782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.089563 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.089570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.089747 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.090255 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.090271 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.092026 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.094863 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.094869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.094873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.095204 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.095548 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.095552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.095556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.095694 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.096189 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.096196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.096317 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.096460 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.096680 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:38:38.097159 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.085361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.3.186.253:43709)
0927 21:38:38.085893 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.085898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.085919 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:38:38.085954 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.085959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.086105 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.086243 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.086911 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.087847 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.088897 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.089385 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.089514 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.092226 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.092235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.092244 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.092272 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.094732 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.095323 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.095330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.095333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.095451 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.095798 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.095802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.095805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.096081 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.096272 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.096590 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.096770 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":291,"threads_started":1}
I20260927 21:38:38.099263 26713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:38.100152 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.099769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50122)
0927 21:38:38.099935 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.100028 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:38:38.100358 26665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
I20260927 21:38:38.101739 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.101439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50130)
0927 21:38:38.101590 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.101652 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 21:38:38.101950 26713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab.
Entry for principal kudu/127.3.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab.
I20260927 21:38:38.110327 3819 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWw8W86/build/asan/bin/kudu
/tmp/dist-test-taskWw8W86/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:45329
--webserver_interface=127.3.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab
--principal=kudu/127.3.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.186.212:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.186.254:40993,127.3.186.253:43709,127.3.186.252:45329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:38:38.151193 26713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 1
I20260927 21:38:38.152196 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.151714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50146)
0927 21:38:38.151905 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.152019 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:38:38.153175 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.152807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50154)
0927 21:38:38.152966 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.153051 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:38:38.153416 26713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
I20260927 21:38:38.221662 26713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 2
I20260927 21:38:38.222668 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.222207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50156)
0927 21:38:38.222407 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.222505 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:38:38.223661 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.223314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50160)
0927 21:38:38.223458 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.223535 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:38:38.223898 26713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
I20260927 21:38:38.296115 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } attempt: 5
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:38:38.312690 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.296881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53424)
0927 21:38:38.297194 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.297217 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.297345 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.298065 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.298073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.298609 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.298881 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.298896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.300390 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.300398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.301883 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.301900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.303614 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.303625 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.303926 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.304697 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.304745 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.306820 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.310660 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.310663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.310666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.310993 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.311372 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.311375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.311377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.311489 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.312037 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.312040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.312153 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.312260 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.312437 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 21:38:38.313041 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.297296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.3.186.253:43709)
0927 21:38:38.297533 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.297542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.297567 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:38:38.297651 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.297657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.297764 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.297917 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.299066 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.300184 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.302034 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.302648 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.302795 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.307012 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.307020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.307027 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.307075 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.310495 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.311107 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.311119 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.311123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.311266 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.311583 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.311587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.311591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.311911 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.312069 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.312408 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.312659 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:38:38.314023 26639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:38.314893 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.314582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50162)
0927 21:38:38.314716 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.314785 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:38:38.315081 26713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 3
I20260927 21:38:38.315847 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.315462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50166)
0927 21:38:38.315634 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.315715 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 21:38:38.315862 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.315514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50174)
0927 21:38:38.315693 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.315758 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 21:38:38.316138 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
I20260927 21:38:38.316881 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.316546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50176)
0927 21:38:38.316734 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.316798 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260927 21:38:38.317088 26713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
I20260927 21:38:38.367372 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 1
I20260927 21:38:38.368399 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.367919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50186)
0927 21:38:38.368118 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.368216 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 21:38:38.369415 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.369044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50196)
0927 21:38:38.369203 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.369294 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:38:38.369696 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
W20260927 21:38:38.415719 26720 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:38:38.415999 26720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:38:38.416112 26720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:38:38.416226 26720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:38:38.423125 26720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:38:38.423192 26720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:38:38.423277 26720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:38:38.423338 26720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:38:38.423434 26720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:38:38.432323 26720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.186.212:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.186.254:40993,127.3.186.253:43709,127.3.186.252:45329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/krb5kdc/kudu_127.3.186.252.keytab
--principal=kudu/127.3.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.186.252:45329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:38:38.433344 26720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:38:38.434510 26720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:38:38.437922 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 2
I20260927 21:38:38.439002 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.438452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50206)
0927 21:38:38.438645 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.438777 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 21:38:38.440201 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.439687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50210)
0927 21:38:38.439948 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.440030 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260927 21:38:38.440469 26639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
W20260927 21:38:38.448072 26725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:38.448026 26728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:38:38.448099 26726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:38:38.449407 26720 server_base.cc:1034] running on GCE node
I20260927 21:38:38.450145 26720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:38:38.451308 26720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:38:38.452534 26720 hybrid_clock.cc:648] HybridClock initialized: now 1790545118452476 us; error 35 us; skew 500 ppm
I20260927 21:38:38.452836 26720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:38:38.455665 26720 init.cc:377] Logged in from keytab as kudu/127.3.186.252@KRBTEST.COM (short username kudu)
I20260927 21:38:38.456902 26720 webserver.cc:460] Webserver started at http://127.3.186.252:41969/ using document root <none> and password file <none>
I20260927 21:38:38.457508 26720 fs_manager.cc:359] Metadata directory not provided
I20260927 21:38:38.457677 26720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:38:38.457948 26720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:38:38.460316 26713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 4
I20260927 21:38:38.460721 26720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data/instance:
uuid: "229fe33afe484263a0accafd241ed7cd"
format_stamp: "Formatted at 2026-09-27 21:38:38 on dist-test-slave-pqr8"
server_key: "b39419d994533afe53372381a11f3945"
server_key_iv: "ffeb145dbbd9545317460e5ba29af3d0"
server_key_version: "encryptionkey@0"
I20260927 21:38:38.461331 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.460818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50216)
0927 21:38:38.461025 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.461121 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 21:38:38.461658 26720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal/instance:
uuid: "229fe33afe484263a0accafd241ed7cd"
format_stamp: "Formatted at 2026-09-27 21:38:38 on dist-test-slave-pqr8"
server_key: "b39419d994533afe53372381a11f3945"
server_key_iv: "ffeb145dbbd9545317460e5ba29af3d0"
server_key_version: "encryptionkey@0"
I20260927 21:38:38.462412 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.462049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50232)
0927 21:38:38.462222 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.462300 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 21:38:38.462605 26713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.186.252:45329: Network error: Client connection negotiation failed: client connection to 127.3.186.252:45329: connect: Connection refused (error 111)
I20260927 21:38:38.466396 26720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:38:38.469687 26735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:38:38.470911 26720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:38:38.471078 26720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data,/tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal
uuid: "229fe33afe484263a0accafd241ed7cd"
format_stamp: "Formatted at 2026-09-27 21:38:38 on dist-test-slave-pqr8"
server_key: "b39419d994533afe53372381a11f3945"
server_key_iv: "ffeb145dbbd9545317460e5ba29af3d0"
server_key_version: "encryptionkey@0"
I20260927 21:38:38.471272 26720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:38:38.490669 26720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:38:38.494089 26720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:38:38.494334 26720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:38:38.511348 26720 rpc_server.cc:307] RPC server started. Bound to: 127.3.186.252:45329
I20260927 21:38:38.511373 26786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.186.252:45329 every 8 connection(s)
I20260927 21:38:38.512392 26720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/data/info.pb
I20260927 21:38:38.515461 26787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:38:38.518996 3819 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWw8W86/build/asan/bin/kudu as pid 26720
I20260927 21:38:38.519181 3819 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWw8W86/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544773327888-3819-0/minicluster-data/master-2/wal/instance
I20260927 21:38:38.519615 3819 external_mini_cluster.cc:1428] Setting key 99be33f3be7910d4791d09ab8b35136f
I20260927 21:38:38.522127 26787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
I20260927 21:38:38.541337 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.525884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.3.186.254:40993)
0927 21:38:38.526077 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.526081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.526723 (+ 642us) server_negotiation.cc:407] Connection header received
0927 21:38:38.526895 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.526899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.526957 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.527027 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.528410 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.529483 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.531317 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.531970 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.532095 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.535597 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.535602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.535605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.535632 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.538785 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.539526 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.539529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.539532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.539666 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.540085 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.540090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.540093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.540410 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.540564 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.540953 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.541091 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:38:38.542057 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.525916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.254:40993 (local address 127.0.0.1:55796)
0927 21:38:38.526557 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.526602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.526772 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.527121 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.527127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.527887 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.528241 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.528258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.529650 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.529654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.531163 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.531175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.532185 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.532190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.532307 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.532894 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.532917 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.535419 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.538902 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.538911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.538917 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.539415 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.539779 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.539785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.539788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.539970 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.540543 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.540550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.540699 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.540873 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.541148 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":297,"threads_started":1}
I20260927 21:38:38.542071 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.521218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.3.186.252:45329)
0927 21:38:38.521930 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.521943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.521989 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:38:38.524051 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.524078 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.524506 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.524883 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.526438 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.528325 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.529979 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.530930 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.531122 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.535086 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.535096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.535108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.535143 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.537919 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.538709 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.538715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.538720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.538902 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.539276 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.539281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.539285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.539828 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.540136 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.540447 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.540825 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":300,"threads_started":1}
I20260927 21:38:38.543417 26787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:38.546685 26639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 3
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.254@KRBTEST.COM
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.254@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:38:38.558998 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.544150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.253:43709 (local address 127.0.0.1:53436)
0927 21:38:38.544370 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.544396 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.544476 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.544780 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.544784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.545110 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.545289 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.545299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.546693 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.546699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.547716 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.547728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.548589 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.548594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.548825 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.549392 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.549412 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.551918 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.555818 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.555825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.555831 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.556416 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.556917 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.556922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.556925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.557102 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.558016 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.558021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.558160 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.558329 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.558605 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 21:38:38.559067 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.544115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.3.186.253:43709)
0927 21:38:38.544305 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.544311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.544454 (+ 143us) server_negotiation.cc:407] Connection header received
0927 21:38:38.544570 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.544575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.544659 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.544761 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.545445 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.546534 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.547847 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.548430 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.548569 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.552094 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.552098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.552101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.552130 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.555638 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.556582 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.556589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.556594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.556769 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.557487 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.557493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.557499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.557873 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.558033 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.558438 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.558644 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:38:38.560340 26787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:38:38.561445 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.545046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.3.186.254:40993)
0927 21:38:38.545226 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.545233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.545254 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:38:38.545296 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.545301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.545365 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.545457 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.546871 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.547868 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.549291 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.549996 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.550114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.554738 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.554746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.554750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.554785 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.558067 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.558837 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.558842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.558846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.558997 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.559451 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.559457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.559460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.559839 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.560128 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.560858 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.561056 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:38:38.562063 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.547230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.3.186.252:45329)
0927 21:38:38.547662 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.547666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.547682 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:38:38.547717 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.547721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.547769 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.547890 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.548649 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.549499 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.551098 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.551698 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.551821 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.555605 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.555631 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.555638 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.555673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.558786 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.559596 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.559600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.559604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.559723 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.560132 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.560136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.560139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.560602 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.560777 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.561614 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.561804 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260927 21:38:38.562887 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.547163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50248)
0927 21:38:38.547389 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.547409 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.547490 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.547921 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.547925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.548336 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.548507 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.548520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.549684 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.549689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.550942 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.550957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.551848 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.551855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.552074 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.552635 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.552678 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.555383 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.558928 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.558932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.558936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.559433 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.559845 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.559850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.559853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.559999 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.560758 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.560765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.560972 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.561176 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.561422 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 21:38:38.568439 26639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9: Bootstrap starting.
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.252@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:38:38.571933 26639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9: Neither blocks nor log segments found. Creating new log.
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.186.253@KRBTEST.COM
I20260927 21:38:38.573160 26639 log.cc:824] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:38.575893 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.561082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.3.186.252:45329)
0927 21:38:38.562260 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.562264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.562280 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:38:38.562321 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.562327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.562387 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.562483 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.563942 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.564701 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.566163 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.566799 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.566906 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.570165 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.570173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.570180 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.570217 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.573073 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.573811 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.573817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.573822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.573943 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.574330 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.574334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.574337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.574711 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.574836 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.575300 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.575512 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":1629,"threads_started":1}
I20260927 21:38:38.576166 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.561017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50264)
0927 21:38:38.561213 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.561228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.561310 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.563116 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.563120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.563593 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.563788 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.563800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.564818 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.564821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.566024 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.566037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.566926 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.566932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.567137 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.567720 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.567742 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.569965 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.573188 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.573194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.573198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.573705 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.574080 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.574086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.574091 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.574243 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.574895 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.574900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.575047 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.575198 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.575643 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":125}
I20260927 21:38:38.576188 26639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9: No bootstrap required, opened a new log
I20260927 21:38:38.580272 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.563106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.3.186.253:43709)
0927 21:38:38.563249 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.563256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.563368 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:38:38.563560 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.563565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.563631 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.563732 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.565161 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.566187 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.567739 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.568335 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.568467 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.573549 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.573554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.573559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.573595 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.576919 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.577735 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.577740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.577743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.577886 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.578278 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.578282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.578285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.578583 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.578859 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.579732 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.579975 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:38:38.580951 26787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: Bootstrap starting.
I20260927 21:38:38.584318 26787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:38.585122 3819 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:38:38.585588 26787 log.cc:824] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:38.585662 26639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.586041 26639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102f6f5c101a48ba8fd5898502134bf9, State: Initialized, Role: FOLLOWER
I20260927 21:38:38.586596 26639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.588219 26614 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "102f6f5c101a48ba8fd5898502134bf9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55798
I20260927 21:38:38.588577 26787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: No bootstrap required, opened a new log
I20260927 21:38:38.590296 26795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.590902 26795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:38.591305 26639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:38.591349 26614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:38:38.591517 26614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:38.595332 26614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.595590 26787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.595950 26787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229fe33afe484263a0accafd241ed7cd, State: Initialized, Role: FOLLOWER
I20260927 21:38:38.596405 26787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.596978 26614 leader_election.cc:288] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [CANDIDATE]: Term 1 election: Requested vote from peers 776a9358b3c1494190439d2b07d1482b (127.3.186.253:43709), 229fe33afe484263a0accafd241ed7cd (127.3.186.252:45329)
I20260927 21:38:38.604923 26799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.605415 26799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:38.607189 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "102f6f5c101a48ba8fd5898502134bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "229fe33afe484263a0accafd241ed7cd"
I20260927 21:38:38.607565 26762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:38.610581 26762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102f6f5c101a48ba8fd5898502134bf9 in term 1.
I20260927 21:38:38.611105 26591 leader_election.cc:302] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102f6f5c101a48ba8fd5898502134bf9, 229fe33afe484263a0accafd241ed7cd; no voters:
I20260927 21:38:38.611497 26787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:38.611497 26795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:38:38.615624 26795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 102f6f5c101a48ba8fd5898502134bf9, State: Running, Role: LEADER
I20260927 21:38:38.616199 26795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.628216 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 102f6f5c101a48ba8fd5898502134bf9. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.628604 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:38.634152 26822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:38:38.641959 26822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:38:38.654084 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.625063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.3.186.254:40993)
0927 21:38:38.625247 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.625252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.628911 (+ 3659us) server_negotiation.cc:407] Connection header received
0927 21:38:38.632322 (+ 3411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.632330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.632422 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.632501 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.634668 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.635739 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.639411 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.640057 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.640146 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.644819 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.644828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.644835 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.644879 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.648752 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.649903 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.649908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.649914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.650079 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.651038 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.651043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.651046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.651842 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.652180 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.653203 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.653541 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:38:38.641047 26572 master_runner.cc:186] Error getting master registration for 127.3.186.253:43709: OK, instance_id { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" instance_seqno: 1790545118046764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:38:38.660199 26572 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:38:38.679910 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.624420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.3.186.253:43709)
0927 21:38:38.624640 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.624649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.628455 (+ 3806us) server_negotiation.cc:407] Connection header received
0927 21:38:38.629192 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.629201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.629300 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.629386 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.636285 (+ 6899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.637326 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.645847 (+ 8521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.646502 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.646606 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.666853 (+ 20247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.666861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.666869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.666917 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.671410 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.674344 (+ 2934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.674352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.674358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.674547 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.675875 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.675883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.675887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.676329 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.676592 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.678445 (+ 1853us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.678950 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:38:38.691169 26762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [term 1 FOLLOWER]: Refusing update from remote peer 102f6f5c101a48ba8fd5898502134bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:38:38.691552 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.628688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.3.186.252:45329)
0927 21:38:38.628841 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.628848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.638920 (+ 10072us) server_negotiation.cc:407] Connection header received
0927 21:38:38.645654 (+ 6734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.645659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.647117 (+ 1458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.647210 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.649014 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.650607 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.662514 (+ 11907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.663336 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.663447 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.683300 (+ 19853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.683305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.683311 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.683351 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.686632 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.687998 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.688003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.688006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.688146 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.689626 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.689632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.689635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.690016 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.690238 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.690913 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.691120 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:38:38.692052 26804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:38:38.695837 26829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:38:38.695950 26829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:38:38.688341 26720 master_runner.cc:186] Error getting master registration for 127.3.186.253:43709: OK, instance_id { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" instance_seqno: 1790545118046764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:38:38.702339 26720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:38:38.719910 26799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 102f6f5c101a48ba8fd5898502134bf9. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.720191 26799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:38.736794 26795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.737135 26795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:38.737944 26822 catalog_manager.cc:1347] Generated new cluster ID: bde5c7499879475e9d1eec8ec8a3f6f2
I20260927 21:38:38.738407 26822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:38:38.738601 26799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.738832 26799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:38.761780 26713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } attempt: 5
I20260927 21:38:38.767378 26822 catalog_manager.cc:1370] Generated new certificate authority record
Sep 27 21:38:38 dist-test-slave-pqr8 krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790545117, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.186.253@KRBTEST.COM for kudu/127.3.186.252@KRBTEST.COM
I20260927 21:38:38.773862 26822 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:38:38.779838 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.762266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.186.252:45329 (local address 127.0.0.1:50284)
0927 21:38:38.762584 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 21:38:38.762602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:38:38.762702 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:38:38.763452 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:38:38.763457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:38:38.763948 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:38:38.764131 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.764142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.765725 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.765738 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.767106 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:38:38.767119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.768092 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.768099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:38:38.768318 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.768990 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:38:38.769014 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:38:38.773083 (+ 4069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:38:38.777152 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.777157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.777161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.777666 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.778045 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:38:38.778049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:38:38.778055 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:38:38.778222 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:38:38.778839 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:38:38.778844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:38:38.778999 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:38:38.779161 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 21:38:38.779444 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260927 21:38:38.782414 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.762885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.3.186.252:45329)
0927 21:38:38.763037 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.763042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.763056 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:38:38.763119 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.763124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.763189 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.763279 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.764330 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.765490 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.767227 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:38.767937 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:38.768054 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:38.773243 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:38:38.773250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:38:38.773256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:38:38.773291 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:38:38.777040 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.777775 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.777780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.777783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.777947 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:38:38.778358 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:38:38.778363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:38:38.778367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:38:38.778705 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:38:38.778882 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:38.781822 (+ 2940us) server_negotiation.cc:299] Negotiation successful
0927 21:38:38.782043 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:38:38.788059 26713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: Bootstrap starting.
I20260927 21:38:38.792954 26713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: Neither blocks nor log segments found. Creating new log.
I20260927 21:38:38.794207 26713 log.cc:824] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:38:38.799062 26713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: No bootstrap required, opened a new log
I20260927 21:38:38.802286 26822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9: Generated new TSK 0
I20260927 21:38:38.803077 26822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:38:38.811594 26713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.811925 26713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776a9358b3c1494190439d2b07d1482b, State: Initialized, Role: FOLLOWER
I20260927 21:38:38.812368 26713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } }
I20260927 21:38:38.814560 26713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:38:38.816675 26687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:38:38.816202 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "102f6f5c101a48ba8fd5898502134bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "776a9358b3c1494190439d2b07d1482b"
I20260927 21:38:38.820859 26687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [term 1 FOLLOWER]: Refusing update from remote peer 102f6f5c101a48ba8fd5898502134bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:38:38.824183 26688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102f6f5c101a48ba8fd5898502134bf9 in term 1.
I20260927 21:38:38.826262 26804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:38:38.813617 26837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.837347 26837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:38.875506 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 102f6f5c101a48ba8fd5898502134bf9. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.875831 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:38:38.876120 26837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:38.876367 26837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:38:38.931128 26642 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:38.927848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.3.186.254:40993)
0927 21:38:38.928056 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.928064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.929319 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 21:38:38.929651 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.929658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.929740 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.929812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.930912 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55822: BlockingRecv error: recv got EOF from 127.0.0.1:55822 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 21:38:38.933176 26717 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:38.927442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.3.186.253:43709)
0927 21:38:38.927669 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.927678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.931727 (+ 4049us) server_negotiation.cc:407] Connection header received
0927 21:38:38.931952 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.931957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.932060 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.932132 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.932988 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53458: BlockingRecv error: recv got EOF from 127.0.0.1:53458 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 21:38:38.939443 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:38.934104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.3.186.252:45329)
0927 21:38:38.934278 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.934284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.934768 (+ 484us) server_negotiation.cc:407] Connection header received
0927 21:38:38.934834 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.934838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.934914 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.935027 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:38.939144 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50298: BlockingRecv error: recv got EOF from 127.0.0.1:50298 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:38:38.957021 26858 catalog_manager.cc:1259] Loaded cluster ID: bde5c7499879475e9d1eec8ec8a3f6f2
I20260927 21:38:38.957154 26858 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: loading cluster ID for follower catalog manager: success
I20260927 21:38:38.960813 26858 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: acquiring CA information for follower catalog manager: success
I20260927 21:38:38.963492 26858 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 776a9358b3c1494190439d2b07d1482b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:38:39.005775 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:38.990041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.3.186.253:43709)
0927 21:38:38.990214 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:38.990220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:38.998436 (+ 8216us) server_negotiation.cc:407] Connection header received
0927 21:38:38.998919 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:38.998926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:38.999104 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:38.999232 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:38.999917 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.001024 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.002996 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.003596 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.003685 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.004338 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.004944 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.005060 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.005300 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.005379 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 21:38:39.048599 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.039660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.3.186.252:45329)
0927 21:38:39.039874 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.039880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.046622 (+ 6742us) server_negotiation.cc:407] Connection header received
0927 21:38:39.047066 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.047077 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.047225 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.047308 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.048350 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50300: BlockingRecv error: recv got EOF from 127.0.0.1:50300 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:38:39.111399 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.097413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.3.186.253:43709)
0927 21:38:39.097621 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.097627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.104641 (+ 7014us) server_negotiation.cc:407] Connection header received
0927 21:38:39.105102 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.105108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.105226 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.105289 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.106012 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.107144 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.109170 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.109913 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.110006 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.110583 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.110748 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.110810 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.111031 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.111110 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 21:38:39.152824 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.144195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.3.186.252:45329)
0927 21:38:39.144376 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.144382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.151082 (+ 6700us) server_negotiation.cc:407] Connection header received
0927 21:38:39.151494 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.151500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.151588 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.151657 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.152575 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50314: BlockingRecv error: recv got EOF from 127.0.0.1:50314 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:38:39.205016 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.191463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.3.186.253:43709)
0927 21:38:39.191637 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.191643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.198405 (+ 6762us) server_negotiation.cc:407] Connection header received
0927 21:38:39.198856 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.198862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.198959 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.199022 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.199683 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.200864 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.202897 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.203535 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.203623 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.204209 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.204330 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.204390 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.204607 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.204690 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 21:38:39.246650 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.237960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.3.186.252:45329)
0927 21:38:39.238153 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.238159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.244859 (+ 6700us) server_negotiation.cc:407] Connection header received
0927 21:38:39.245271 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.245277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.245373 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.245450 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.246427 (+ 977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50318: BlockingRecv error: recv got EOF from 127.0.0.1:50318 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:38:39.266566 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.259423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.3.186.253:43709)
0927 21:38:39.259624 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.259630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.260115 (+ 485us) server_negotiation.cc:407] Connection header received
0927 21:38:39.260258 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.260263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.260351 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.260451 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.260912 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.262071 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.263931 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.264630 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.264753 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.265824 (+ 1071us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.265948 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.266043 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.266207 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.266302 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 21:38:39.274309 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.272199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.3.186.252:45329)
0927 21:38:39.272448 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.272454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.272962 (+ 508us) server_negotiation.cc:407] Connection header received
0927 21:38:39.273110 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.273115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.273196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.273295 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.274084 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50332: BlockingRecv error: recv got EOF from 127.0.0.1:50332 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260927 21:38:39.290755 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.285390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.3.186.253:43709)
0927 21:38:39.285618 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.285623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.285907 (+ 284us) server_negotiation.cc:407] Connection header received
0927 21:38:39.286047 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.286050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.286112 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.286182 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.286543 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.287252 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.289099 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.289503 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.289665 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.290066 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.290167 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.290228 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.290381 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.290433 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 21:38:39.297701 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.295901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.3.186.252:45329)
0927 21:38:39.296051 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.296057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.296439 (+ 382us) server_negotiation.cc:407] Connection header received
0927 21:38:39.296554 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.296559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.296640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.296770 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.297442 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50346: BlockingRecv error: recv got EOF from 127.0.0.1:50346 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:38:39.322139 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.316308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.3.186.253:43709)
0927 21:38:39.316488 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.316494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.316837 (+ 343us) server_negotiation.cc:407] Connection header received
0927 21:38:39.316980 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.316985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.317071 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.317174 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.317508 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.318765 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.320151 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.320888 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.321014 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.321438 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.321608 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.321717 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.321794 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.321872 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 21:38:39.330521 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.327853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.3.186.252:45329)
0927 21:38:39.329011 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.329019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.329039 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:38:39.329104 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.329111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.329209 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.329310 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.330289 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50362: BlockingRecv error: recv got EOF from 127.0.0.1:50362 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:38:39.371524 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.365672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.3.186.253:43709)
0927 21:38:39.365882 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.365888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.366325 (+ 437us) server_negotiation.cc:407] Connection header received
0927 21:38:39.366524 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.366531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.366677 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.366836 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.367272 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.368337 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.369659 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.370396 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.370519 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.370832 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.370954 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.371051 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.371140 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.371217 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 21:38:39.378619 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.376925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.3.186.252:45329)
0927 21:38:39.377091 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.377097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.377409 (+ 312us) server_negotiation.cc:407] Connection header received
0927 21:38:39.377542 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.377550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.377650 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.377789 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.378412 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50370: BlockingRecv error: recv got EOF from 127.0.0.1:50370 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:38:39.420082 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "102f6f5c101a48ba8fd5898502134bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f6f5c101a48ba8fd5898502134bf9" member_type: VOTER last_known_addr { host: "127.3.186.254" port: 40993 } } peers { permanent_uuid: "776a9358b3c1494190439d2b07d1482b" member_type: VOTER last_known_addr { host: "127.3.186.253" port: 43709 } } peers { permanent_uuid: "229fe33afe484263a0accafd241ed7cd" member_type: VOTER last_known_addr { host: "127.3.186.252" port: 45329 } } }
I20260927 21:38:39.420447 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102f6f5c101a48ba8fd5898502134bf9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:38:39.454780 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.449054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.3.186.253:43709)
0927 21:38:39.449257 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.449264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.449714 (+ 450us) server_negotiation.cc:407] Connection header received
0927 21:38:39.449880 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.449886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.449979 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.450078 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.450436 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.451585 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.452838 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.453471 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.453654 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.454056 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.454208 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.454290 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.454414 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.454488 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 21:38:39.461786 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.460048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.3.186.252:45329)
0927 21:38:39.460253 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.460259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.460575 (+ 316us) server_negotiation.cc:407] Connection header received
0927 21:38:39.460708 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.460714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.460820 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.460924 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.461582 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50382: BlockingRecv error: recv got EOF from 127.0.0.1:50382 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:38:39.597335 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.592767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.3.186.253:43709)
0927 21:38:39.593000 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.593007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.593231 (+ 224us) server_negotiation.cc:407] Connection header received
0927 21:38:39.593382 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.593385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.593453 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.593605 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.593924 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.594597 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.595782 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.596161 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.596248 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.596705 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.596808 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.596874 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.597025 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.597085 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 21:38:39.604789 26790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:38:39.602955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.3.186.252:45329)
0927 21:38:39.603106 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.603110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.603404 (+ 294us) server_negotiation.cc:407] Connection header received
0927 21:38:39.603576 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.603593 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.603661 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.603737 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:38:39.604525 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50386: BlockingRecv error: recv got EOF from 127.0.0.1:50386 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:38:39.697986 26829 catalog_manager.cc:1259] Loaded cluster ID: bde5c7499879475e9d1eec8ec8a3f6f2
I20260927 21:38:39.698073 26829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: loading cluster ID for follower catalog manager: success
I20260927 21:38:39.700479 26829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: acquiring CA information for follower catalog manager: success
I20260927 21:38:39.702243 26829 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 229fe33afe484263a0accafd241ed7cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:38:39.868964 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.863896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.3.186.253:43709)
0927 21:38:39.864080 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.864085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.864300 (+ 215us) server_negotiation.cc:407] Connection header received
0927 21:38:39.864445 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.864448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.864530 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.864611 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.864975 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.865769 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.867253 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.867667 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.867832 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.868249 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.868340 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.868434 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.868573 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.868646 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:38:39.879215 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.874175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.3.186.252:45329)
0927 21:38:39.874383 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.874388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.874531 (+ 143us) server_negotiation.cc:407] Connection header received
0927 21:38:39.874694 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.874700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.874860 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.874953 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.875333 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.876000 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.877177 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.877565 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.877659 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.878088 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.878597 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.878721 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.878820 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.878880 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:38:39.891464 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.884540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.3.186.254:40993)
0927 21:38:39.885098 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.885107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.885693 (+ 586us) server_negotiation.cc:407] Connection header received
0927 21:38:39.886050 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.886059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.886281 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.886398 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.887094 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.887869 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.889585 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.890006 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.890139 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.890595 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.890882 (+ 287us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.891006 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.891131 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.891209 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
I20260927 21:38:39.893501 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.884573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.3.186.253:43709)
0927 21:38:39.884728 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.884733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.886599 (+ 1866us) server_negotiation.cc:407] Connection header received
0927 21:38:39.886798 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.886804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.886886 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.886979 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.887658 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.888613 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.891210 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.891802 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.891880 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.892780 (+ 900us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.892876 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.892971 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.893115 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.893185 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:38:39.895803 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26572
I20260927 21:38:39.896762 26634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:39.896880 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:38:39.885782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.3.186.252:45329)
0927 21:38:39.885968 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:38:39.885976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:38:39.891143 (+ 5167us) server_negotiation.cc:407] Connection header received
0927 21:38:39.891570 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:38:39.891575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:38:39.891671 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:38:39.891779 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:38:39.892245 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.893079 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.895119 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:38:39.895629 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:38:39.895717 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:38:39.896176 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:38:39.896295 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:38:39.896372 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:38:39.896523 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 21:38:39.896598 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:38:40.001390 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26572
I20260927 21:38:40.021359 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26646
I20260927 21:38:40.022389 26708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:40.124003 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26646
I20260927 21:38:40.143911 3819 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWw8W86/build/asan/bin/kudu pid 26720
I20260927 21:38:40.144867 26782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:38:40.244774 3819 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWw8W86/build/asan/bin/kudu with pid 26720
2026-09-27T21:38:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4313 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8904 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (346887 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST